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SUMMARY 


This  report  is  the  first  part  of  a  project  to  evaluate  how  the  Integrated  Maintenance 
Information  System  (BVUS)  can  be  used  in  aircraft  maintenance  training.  The  report  presents  the 
results  of  a  Training  Situation  Analysis,  in  which  the  current  Air  Force  maintenance  training 
environment  was  analyzed  and  h5^otheses  were  developed  about  how  IMIS  may  affect 
maintenance  training.  In  order  to  develop  hypotheses  about  IMIS's  effect  on  training,  the  Air 
Force  maintenance  process  and  IMIS's  capabilities  were  also  investigated. 

In  performing  the  Training  Situation  Analysis,  structured  interviews  were  conducted  at  Air 
Force  maintenance  and  maintenance  training  facihties,  including  Technical  School,  Field 
Training  Detachment,  Logistics  Support  Training,  and  on-the-job-training  sites.  Additional  data 
were  collected  through  literature  reviews  and  observations  of  IMIS  demonstrations. 

The  data  suggested  that  maintenance  technicians  need  training  in  1)  a  variety  of  maintenance 
procedures,  and  2)  the  problem-solving  skills  needed  to  handle  troubleshooting  problems  that  are 
not  solvable  by  standardized  procedures  (e.g.,  technical  orders  (TOs)).  Troubleshooting 
problems  that  require  problem-solving  knowledge  beyond  that  in  the  TOs  are  fairly  common  in 
the  Air  Force  maintenance  environment.  Current  training  to  provide  this  problem-solving 
knowledge  needs  considerable  improvement.  IMIS  can  help  provide  an  effective  solution  to  this 
training  problem.  Research  suggests  that  IMIS  can  be  used  most  efficiently  as  a  simulator  in 
which  the  intelligence  and  knowledge  of  the  IMIS  Diagnostic  Module  is  used  to  provide 
coaching  and  practice  on  troubleshooting  problems. 

Part  two  of  this  research  project  entails  a  demonstration  of  some  of  the  training  capabilities  of 
IMIS  discussed  in  this  report.  This  demonstration  will  make  use  of  the  IMIS  Portable 
Maintenance  Aid  and  other  components  to  enhance  training  of  Air  Force  maintenance 
technicians. 
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TRAINING  SITUATION  ANALYSIS 
FOR  FLIGHTLINE  MAINTENANCE  TRAINING: 
IMPLICATIONS  FOR  IMIS 

1.  INTRODUCTION 

The  Air  Force  Armstrong  Laboratory  is  conducting  research  to  develop  and  demonstrate  a 
prototype  Integrated  Maintenance  Information  System  (IMIS).  IMIS  is  a  job  aid  that  will  help 
technicians  in  many  aspects  of  maintenance  activities.  The  prototype  IMIS  will  allow  access  to, 
and  integration  of,  information  for  such  activities  as  flight  debriefing,  opening  and  closing  work 
orders,  interactive  diagnostics,  review  of  technical  manuals,  parts  ordering,  automatic 
maintenance  data  collection,  and  tracking  aircraft  status.  These  automated  capabilities  are 
expected  to  greatly  improve  the  way  flightline  maintenance  technicians  perform  their  jobs.  IMIS 
is  currently  being  developed  for  maintenance  of  the  Air  Force's  F-16,  F-22,  and  Navy's  F/A-18 
aircraft. 

While  the  potential  of  IMIS  as  a  viable  job  aid  has  long  been  recognized,  the  concept  of  using 
IMIS  to  effectively  present  training  has  only  begun  to  be  explored  (Babiarz,  Jemigan  & 
Gumienny,  1989;  Brandt,  Jemigan  &  Dierker,  1987).  Mei  Technology  Corporation,  under 
contract  to  the  Air  Force  Armstrong  Laboratory,  Human  Resources  Directorate,  undertook  a 
research  effort  to  determine  whether  IMIS  holds  promise  for  training,  what  specific  instructional 
activities  can  best  make  use  of  IMIS'  capabilities,  and  what  changes,  if  any,  are  necessary  to 
allow  IMIS  to  fully  implement  a  training  function.  This  report  describes  the  first  of  several 
major  steps  taken  in  analyzing  IMIS  training  capabilities  --  conducting  a  Training  Situation 
Analysis  for  maintenance  training.  Our  main  goals  in  this  report  were  to  (1)  assess  the  current 
Air  Force  maintenance  training  situation  and  establish  a  baseline  for  further  research,  and  (2) 
develop  hypotheses  about  the  effect  IMIS  will  have  on  maintenance  training.  However,  in  order 
to  understand  IMIS's  effect  on  training,  information  was  needed  on  more  than  the  current  traimng 
situation.  We  also  needed  to  understand  the  nature  of  the  Air  Force  maintenance  process  itself, 
since  planning  computer-based  training  for  a  task  requires  detailed  knowledge  of  how  that  task  is 
performed.  In  addition,  we  had  to  obtain  information  about  IMIS's  capabilities.  Therefore,  in 
addition  to  investigating  the  current  training  situation,  we  also  gathered  data  on  the  maintenance 
process  and  IMIS's  capabilities. 

Scope 

Establishing  a  maintenance  training  baseline  was  necessary  to  determine  how  the  application 
of  IMIS  could  increase  the  effectiveness  of  both  on-the-job  and  classroom  training.  Therefore,  in 
addition  to  studying  on-the-job  training  (OJT)  practices  at  operational  Air  Force  bases,  we 
investigated  maintenance  training  practices  at  Technical  Training  Centers  (Technical  Schools), 
where  new  airmen  report  after  basic  training,  and  Field  Training  Detachments  (FTDs),  where 
additional  maintenance-related  training  is  provided  at  operational  bases. 

Our  analysis  centered  on  information  relating  to  maintenance  of  five  F-16  sub-systems, 
namely,  the  Fire  Control  Radar,  Hydraulic  Power  Supply,  Heads-Up  Display,  Cabin 
Pressurization  System,  and  Inertial  Navigating  System.  These  five  sub-systems  were  previously 
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selected  by  the  Armstrong  Laboratory  as  part  of  the  IMIS  prototype  system  development.  The 
analysis  considered  such  factors  as  the  specific  tasks  trained,  who  conducts  the  training,  how 
training  takes  place,  the  duration  of  training,  how  individual  training  accomplishments  are 
recorded,  and  trainee  reliance  on  job  aids  of  various  kinds,  in  particular  the  technical  orders 
(TOs)  and  other  documentation.  In  other  words,  we  attempted  to  provide  a  representative 
description  of  current  practices  in  OJT  and  during  formal  training  in  the  Technical  Schools  and 
FTD  classrooms. 

In  the  remainder  of  this  report,  we  first  describe  the  methodology  used  for  the  Training 
Situation  Analysis,  then  summarize  some  important  findings  from  a  literature  review  of 
maintenance  and  maintenance  training  issues.  These  findings  are  used  in  interpreting  the  data 
from  the  Training  Situation  Analysis.  Following  this,  results  of  the  Training  Situation  Analysis 
are  presented  in  three  sections,  focusing  on  IMIS's  capabilities,  the  Air  Force  maintenance 
environment,  and  the  Air  Force  maintenance  training  environment.  Finally,  we  develop  and 
describe  hypotheses  about  how  IMIS  may  affect  maintenance  training. 

II.  METHODOLOGY 

Observations  of  IMIS 

To  gather  data  on  IMIS's  capabihties,  we  followed  a  three-step  process  consisting  of:  a 
literature  review  of  all  IMIS  components;  briefings  and  direct  observation  of  an  F/A-18  IMIS 
component  demonstration  at  the  Marine  Corps  Air  Station,  Beaufort,  SC;  and  meetings  with 
IMIS  experts  ait  Wright-Patterson  AFB,  OH.  This  information  was  used  in  developing 
hypotheses  as  to  how  IMIS  could  be  used  to  improve  or  supplement  current  Air  Force  training. 

First,  technical  reports  were  reviewed  for  general  information  concerning  various 
components  of  IMIS  such  as  the  Portable  Maintenance  Aid  (PMA),  Authoring  and  Presentation 
System  (APS),  Content  Data  Model  (CDM),  Diagnostics  Module  (DM),  and  the  Maintenance 
Workstation.  We  obtained  information  on  IMIS  from  a  number  of  reports  (Babiarz,  et  al.,  1989; 
Brandt,  et  al.,  1987;  Link,  Von  Holle,  &  Mason,  1987;  Cooke,  Jemigan,  Huntington,  Myers, 
Gumienny,  &  Maiorana,  1990;  Cooke,  Jemigan,  Maiorana,  &  Myers,  1990;  Cooke,  Jemigan, 
Myers,  &  Maiorana,  1990;  Cooke,  Maiorana,  Myers,  &  Jemigan,  1991;  Earl,  Gunning,  Freese, 
Shroder,  Werts,  &  Reser,  1990;  Link,  Murphy,  Carlson,  Thomas,  Brown,  Sc  Joyce,  1990).  These 
reports  contain  considerable  information  about  IMIS,  but  do  not  describe  the  details  of  the  most 
recent  versions  of  EVUS.  However,  the  reports  provided  the  analysts  with  a  knowledge  base  that 
could  then  be  enhanced  through  observations  of  the  various  IMIS  components. 

Later,  through  meetings  with  Logistics  Research  Division  personnel  at  Wright-Patterson 
AFB,  analysts  clarified  any  questions  concerning  the  various  components  of  IMIS,  how  they 
operie  together,  and  ways  in  which  they  might  affect  the  Air  Force  maintenance  community. 
Although  these  meetings  and  observations  yielded  valuable  information,  certain  issues  arose.  In 
particular,  at  the  time  of  our  site  visits,  the  APS  and  the  PMA  for  the  F-16  were  still  being 
developed  and  revised.  Analysts  viewed  the  F/A-18  PMA  and  APS  in  use  during  observations  at 
the  Marine  Corps  Air  Station,  Beaufort,  SC  and  Wright-Patterson  AFB,  OH  respectively.  Unlike 
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the  F/A-18,  the  APS  for  the  F-16  is  expected  to  have  a  more  user-friendly  format.  The  PMA  for 
the  F-16  will  weigh  less,  will  have  a  different  keypad,  and  will  be  menu-driven  to  a  greater 
extent.  Because  we  were  able  to  view  only  F/A-18  components  and  partially  discuss  F-16 
components  during  the  visits,  the  information  necessary  to  form  hypotheses  about  specific  IMIS 
effects  on  F-16  training  was  somewhat  limited.  For  example,  when  developing  hypotheses  about 
the  IMIS  capabilities  to  present  remedial  training,  it  is  necessary  to  know  whether  the  F-16 
version  of  the  PMA  will  allow  an  expert  to  receive  novice  level  help  (and  vice  versa),  as  needed, 
at  any  point  in  the  lesson.  Currently  with  the  F/A-18,  the  technician  must  return  to  the  beginning 
of  the  task  rather  than  receive  remedial  assistance  in  the  middle  of  a  work  order.  This  implies 
that  if  the  technician  is  in  the  expert  mode  and  wants  to  receive  novice  level  help  at  the  task  level, 
he  or  she  would  have  to  get  out  of  the  task,  go  back  to  the  beginning,  and  start  over  at  a  novice 
level,  repeating  much  already-mastered  task  information. 

Observation  of  Maintenance  and  Maintenance  Training  Process 

Data  on  both  the  maintenance  and  the  maintenance  training  processes  were  obtained  by 
structured  interviews  at  the  Kelly  AFB,  Texas,  Hill  AFB,  Utah,  and  Lowry  AFB,  Colorado. 
Therefore,  the  methodologies  for  obtaining  these  data  are  explained  together.  Below,  we 
describe  our  methods  for  selecting  Air  Force  Specialties  (AFSs)  and  tasks,  developing  the 
questionnaire,  and  performing  the  situation  analysis. 

Selecting  the  Air  Force  Specialties 

While  IMIS  is  aimed  at  the  entire  spectrum  of  maintenance,  only  a  small  segment  of  the 
maintenance  community  was  included  in  the  IMIS  Prototype  Field  Test  scheduled  for  spring  of 
1993.  The  five  F-16  sub-systems  mentioned  earlier  were  recommended  for  use  by  the  Air  Force 
Armstrong  Laboratory  in  the  Prototype  Field  Test,  because  they  gave  the  broadest  aircraft 
maintenance  coverage.  These  were  also  considered  to  be  the  systems  where  IMIS  potentially 
would  have  the  greatest  effect  on  both  classroom  and  on-the-job  training.  In  line  with  this,  we 
reviewed  all  AFSs  involved  in  aircraft  maintenance  as  potential  candidates  for  further  research. 
A  list  was  generated  of  eight  AFSs  that  are  involved  in  the  maintenance  of  the  above  mentioned 
subsystems.  Using  the  following  criteria,  three  AFSs  were  chosen  as  candidates  for  further 
research. 

Subsystem  Coverage.  The  strongest  candidate  AFSs  were  those  whose  tasks  address  the 
largest  majority  of  the  five  F-16  subsystems.  In  other  words,  it  was  necessary  to  identify  the 
smallest  number  of  AFSs  containing  tasks  to  cover  maintenance  of  all  five  subsystems.  Analysts 
found  that  there  was  not  a  one-to-one  AFS-subsystem  match  for  all  subsystems,  nor  was  there  a 
single  AFS  that  supported  or  covered  all  five  subsystems.  Three  AFSs  resulted  from  this 
analysis;  they  were:  Air  Force  Specialty  Code  (AFSC)  452X2A,  F-16  Avionic  Systems,  (which 
covered  three  of  the  subsystems),  AFSCs  452X5,  Tactical  Electrical  and  Environmental  Systems, 
and  454X4/A,  Aircraft  Pneudraulic  Systems  (each  of  which  covered  the  remaining  two 
subsystems).  Thus  all  five  F-16  sub-systems  selected  for  the  IMIS  test  were  covered  by  three 
AFSCs. 
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Recency.  The  strongest  candidate  AFSs  were  those  with  most  recently  dated  Occupational 
Survey  Report  (OSR)  and  Specialty  Training  Standard  (STS)  data.  Analysts  chose  these  criteria 
realizing  that  OSR  data  could  change  within  a  3-  to  5-year  time  period.  And  even  then  the  most 
recent  OSR  data  could  still  be  out  of  date  depending  upon  recently  occurring  changes.  For 
instance,  it  is  possible  for  the  conduct  of  maintenance  for  certain  tasks  to  change  due  to  changes 
in  TOs  and  this  information  not  be  included  in  the  last  occupational  survey  data  collection 
procedures.  In  order  to  have  the  most  accurate,  relevant  information  for  later  decisions  and 
research,  the  most  recent  data  were  needed. 


Table  1 

Maintenance  Tasks  Used  in  Training  Situation  Analysis 


AFSC 

TASKS 

452X2/A:  Avionic  Systems 

1.  Isolate  malfunctions  to  inertial  navigation  line- 
replaceable  units. 

2.  Perform  fire  control  radar  integration  checks. 

3.  Isolate  malfunctions  to  head-up  display  pilot  units. 

452X5;  Tactical  Electrical  & 

Environmental  Systems 

1.  Perform  leakage  checks  of  cabin  or  cargo 
pressurization  systems. 

2.  Perform  operational  checks  of  cabin  or  cargo 
pressurization  systems. 

3.  Isolate  malfunctions  to  cabin  or  cargo 
pressurization  systems. 

454X4/A:  Aircraft  Pneudraulic 
Systems 

1.  Perform  operational  checks  of  hydraulic  power 
systems  /  Isolate  malfunctions  within  hydraulic 
systems  using  hydraulic  schematics. 

2.  Inspect  aircraft  installed  hydraulic  power  systems. 

3.  Remove  or  install  components  of  hydraulic  power 
systems. 

Selecting  the  Tasks 

Specific  task  information  for  the  selected  AFSs  was  extracted  from  the  OSR  data  along  with 
accompanying  Training  Emphasis  (TE)  and  Task  Difficulty  (TD)  ratings.  Tasks  were  chosen  that 
met  or  exceeded  the  OSR-reported  "high"  rating  for  both  TE  and  TD,  and  had  high  scores  on 
percent  members  performing  and  relative  time  spent  on  task  ratings.  The  assumption  here  was 
that  tasks  selected  for  MIS  demonstration,  analysis,  and  hypothesis  testing  should  be  sufficiently 
rigorous  in  that  they  require  (1)  a  large  amount  of  training,  (2)  lengthy  time  to  learn  to  perform, 
and  (3)  performance  by  the  largest  number  of  technicians.  The  individual  tasks  were  then  rank- 
ordered  according  to  highest  combined  TE,  TD  and  percent  members  performing  ratings. 

STS  data  for  the  relevant  AFSs  was  reviewed  in  order  to  verify  whether  changes  had  taken 
place  in  the  training  of  the  selected  tasks.  These  data  were  also  reviewed  to  verify  whether  the 
tasks  themselves  were  still  relevant.  The  final  selection  of  tasks  to  be  used  as  the  basis  for  the 
Training  Situation  Analysis  was  made  by  maintenance  experts  at  Hill  AFB,  in  each  of  the  three 
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AFSs.  These  experts  were  asked  to  choose  three  tasks  and  rank  order  them  according  to  those 
that  were  most  frequently  performed,  and  those  for  which  personnel  would  have  the  most 
information  to  offer.  Respondents  referred  to  these  tasks,  which  are  listed  in  Table  1,  when 
answering  the  questionnaire. 

Develoning  the  Questionnaire 

To  better  estimate  the  impact  IMIS  would  have  on  the  training  of  various  tasks,  we  developed 
an  interview  questionnaire  to  assess  the  current  training  situation  in  various  environments  (OJT, 
FTD,  and  Technical  School).  Information  obtained  from  OSR  and  STS  data,  and  from  a 
literature  review  on  maintenance  and  troubleshooting  processes,  was  used  to  develop  the 
questionnaire.  Along  with  this,  a  decision  criteria  checklist  for  training/job  aiding  tradeoffs  was 
developed  from  which  statements  were  extracted  for  use  in  the  questionnaire.  These  decision 
criteria  were  developed  according  to  information  obtained  from  research  conducted  on 
training/job  aiding  tradeoff  decisions  (Harless,  1979;  Smillie,  1985;  Zenyuh,  Frey,  Rouse  & 
Lamb,  1991).  It  was  hoped  that  responses  to  these  statements  would  enable  the  analysts  to 
determine  whether  use  of  IMIS  for  particular  tasks  was  more  appropriate  as  a  job  aid  or  as  a 
training  device.  These  results  will  be  discussed  in  a  later  section  (Findings  Concerning  Task 
Characteristics). 

The  interview  questionnaire  varied  in  length  and  content  according  to  the  job  category  of  the 
personnel  being  interviewed.  The  questionnaire  elicited  information  on  namely  task  performance 
status,  task  characteristics  as  they  relate  to  flightline  maintenance,  training  issues,  and 
troubleshooting.  Different  versions  of  the  questionnaire  were  developed  in  order  to  obtain  the 
most  appropriate  types  of  information  from  all  personnel  involved.  For  instance,  questions  posed 
to  maintenance  technicians  were  performance  oriented,  while  the  maintenance  training  manager 
received  questions  placing  greater  emphasis  on  maintenance  training.  Specific  tasks  (see  Table 
1)  were  included  at  the  beginning  of  each  questionnaire.  Respondents  were  asked  to  refer  to 
these  tasks  while  answering  all  questions.  The  same  questions  were  repeated  for  each  task  to 
determine  variation  across  tasks.  A  supplemental  set  of  questions  pertaining  to  troubleshooting 
performance  and  training  was  also  included.  These  questions  did  not  address  a  specific  task  and 
were  answered  only  once  by  each  respondent.  One  questionnaire  is  presented  in  Appendix  A. 

Assessing  the  Maintenance  and  Training  Situation 

Interviews  were  conducted  with  maintenance  supervisors,  maintenance  technicians, 
maintenance  training  managers,  and  FTD  Instmctors  within  operational  units  at  Kelly  AFB,  TX 
and  Hill  AFB,  UT.  An  informal  briefing  was  first  held  at  the  149th  Tactical  Fighter  Group, 
Texas  Air  National  Guard  unit  at  Kelly  AFB  concerning  the  use  of  IMIS  for  maintenance  and 
maintenance  training  activities.  Structured  interviews  were  then  conducted  with  relevant 
personnel  in  order  to  pilot  test  the  questionnaire.  Based  on  feedback  from  the  pilot  testing,  minor 
changes  were  made  in  the  questionnaires  before  using  them  for  the  interviews  at  Hill  AFB. 

Additionally,  interviews  were  conducted  with  technical  school  instructors,  a  course  director, 
and  a  training  developer  at  the  Technical  Training  Center  at  Lowry  AFB,  CO.  It  should  be  noted 
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that  training  for  only  one  of  the  three  AFSs  chosen  for  this  project~Av/onic5/¥52Z2--occurs  at 
this  installation.  However,  this  particular  AFS  provides  coverage  for  three  of  the  five  F-16 
subsystems  of  interest  as  noted  earlier.  It  was  not  feasible  to  interview  technical  school 
personnel  for  the  two  remaining  AFSs. 

Identification  of  the  best  qualified  persons  to  be  interviewed  was  accomplished  using  expert 
(nonprobability)  sampling  procedures.  An  Air  Force  point  of  contact  was  established  at  each 
base/Technical  Training  Center  to  arrange  interviews.  We  specified  criteria  for  selecting 
interviewees  as  shown  in  Appendix  B. 

Each  respondent  filled  out  the  questionnaire  during  a  structured-interview  session  with  an 
individual  analyst.  This  allowed  the  respondents  to  easily  ask  questions  about  the  questionnaire 
and  verbally  elaborate  on  their  answers. 

in.  TRAINING  SITUATION  ANALYSIS  FINDINGS 

3.1.  Findings  from  the  Literature  Review 

The  literature  review  focused  on  two  research  topics,  the  maintenance  process  and  the 
maintenance  training  process.  The  findings  from  the  literature  review  were  used  in  generating 
some  of  the  questions  for  the  stmctured  interviews,  and  also  in  interpreting  the  data.  A  few  of 
the  important  points  from  the  literature  review  are  summarized  here. 

3.1.1.  Knowledge  Used  In  Maintenance  and  Troubleshooting. 

Research  by  cognitive  psychologists  has  shown  that  people  who  are  expert  at  a  particular  task 
possess  a  variety  of  different  kinds  of  task  related  knowledge,  from  domain-specific  procedures 
to  general  problem-solving  strategies.  Much  of  this  knowledge  is  tacit,  i.e.,  not  easily  available 
for  conscious  discussion  by  the  expert.  An  effective  way  to  train  students  on  a  task  is  to  map  out 
the  repertoire  of  knowledge  used  by  the  expert  and  explicitly  teach  this  to  students  (Gott,  1988). 
Recently,  a  number  of  psychologists  have  studied  the  complex  process  of  maintenance  and 
troubleshooting,  and  have  identified  some  of  the  key  knowledge  used  in  these  difficult  tasks 
(Gott,  1988;  Gugerty,  1989;  Lesgold  &  Lajoie,  1991;  Rasmussen,  1981).  This  knowledge  is 
described  below.  Understanding  it  will  help  in  determining  how  IMIS  could  be  used  affectively 
in  maintenance  training. 

Most  maintenance  tasks,  like  tasks  at  other  jobs,  involve  following  pre-leamed  procedures. 
For  example,  a  maintenance  technician  must  follow  safety  and  administrative  procedures,  as  well 
as  procedures  for  using  tools  and  test  equipment  and  for  removing  and  replacing  components. 
An  experienced  technician  can  perform  these  procedures  easily.  In  contrast  to  procedure¬ 
following  tasks  are  troubleshooting  tasks,  in  which  the  maintenance  technician  must  find  and  fix 
the  cause  of  a  malfunction  in  a  system  (e.g.,  an  aircraft).  Troubleshooting  tasks  can  be  very 
difficult,  sometimes  challenging  the  most  experienced  technicians  to  do  complex  problem 
solving.  Errors  in  troubleshooting  contribute  significantly  towards  the  25%  to  30%  of  the 
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military  budget  spent  on  maintenance  (Means  &  Gott,  1988).  Because  of  the  difficulty  and 
importance  of  troubleshooting  tasks,  we  focused  on  these  tasks  in  our  literature  review. 

Table  2  lists  the  key  knowledge  required  for  expert  troubleshooting  under  two  categories, 
declarative  (or  factual)  knowledge  and  procedural  (or  skill)  knowledge.  The  procedural 
knowledge  (or  cognitive  processes)  is  hierarchical  in  nature,  with  the  coordination  processes 
being  the  most  general.  These  processes  organize  the  use  of  strategies,  which  in  turn  control  the 
use  of  the  more  specific  procedures.  There  is  no  hierarchical  relationship  between  the  two  types 
of  declarative  knowledge,  mental-model  and  symptom-fault-association  knowledge.  Rather, 
each  of  these  types  of  knowledge  is  used  by  different  troubleshooting  strategies,  as  is  described 
below. 

Mental  model  knowledge,  in  the  context  of  troubleshooting,  is  knowledge  of  how  the  device 
or  system  works.  It  includes  knowledge  of  the  internal  structure  (or  topography)  of  the  system, 
the  functions  of  system  components,  and  the  states  (or  behavior)  of  the  system.  A  mental  model 
can  be  used  to  answer  what-if  questions  about  the  system,  such  as  how  the  failure  of  a  particular 
component  will  affect  the  behavior  of  the  rest  of  the  system.  Symptom-fault  associations  are 
remembered  associations  between  specific  symptoms  (improper  system  behaviors)  and  the 
internal  system  faults  that  usually  cause  them.  When  your  mechanic  diagnoses  your  car  problem 
instantly  after  listening  to  the  engine  and  looking  at  the  color  of  the  exhaust  smoke,  he  or  she  is 
using  symptom-fault  associations. 

Table  2 

Knowledge  Used  in  Maintenance  and  Troubleshooting 


Knowledge 

Category 

Knowledge  Type 

Description 

Mental  Models 

"How-it-works"  knowledge  pertaining  to  system 
structure  (topography),  function,  and  behavior. 

Declarative 

Symptom-fault 

Associations  between  symptoms  and  faults.  Fault 

Associations 

probabilities. 

Procedures 

Step-by-step  information  about  how  to  perform 
specific  actions 

Procedural 

(Cognitive 

Strategies 

More  general  processes  that  coordinate  the  use  of 

Processes) 

procedures,  using  mental  models  or  symptom-fault 
associations. 

. 

Coordination 

Very  general,  metacognitive  processes  that  involve 

Processes 

activities  such  as  strategy  selection. 
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Procedural  knowledge  consists  of  step-by-step  information  about  how  to  perform  specific 
actions,  such  as  testing  resistance  with  a  multimeter.  Strategies  are  more  general  processes  that 
help  organize  technicians'  search  for  the  faulty  component.  There  are  two  main  kinds  of 
troubleshooting  strategies  (Rasmussen,  1981).  The  first  involves  using  symptom-fault- 
association  knowledge  to  recall  the  fault  that  was  found  to  cause  a  symptom  in  the  past,  and  then 
testing  for  this  fault.  The  second  involves  using  mental-model  knowledge.  An  example  of  the 
second  type  of  strategy  is  elimination,  in  which  the  technician  uses  information  about  correct 
system  behaviors  and  knowledge  of  system  function  and  topography  to  eliminate  from 
consideration  components  that  can  be  inferred  to  be  working.  So,  if  your  car  will  not  start  but 
your  lights  work,  you  can  eliminate  the  battery  as  a  possible  cause  of  the  problem. 

Coordination  processes  involve  metacognitive  thinking,  in  which  technicians  monitor  how 
well  their  strategies  are  meeting  the  various  constraints  of  the  problem  (e.g.,  finding  the  fault, 
having  a  plane  ready  on  time)  and  change  strategies  accordingly.  A  technician  is  using 
coordination  processes  when  he  or  she  decides,  because  of  time  constraints,  to  stop  trying  to 
isolate  (locate)  a  specific  fault  and  instead  make  a  costly  component  replacement  that  will  fix  the 
problem  quickly. 

3.1.2.  Maintenance  Training  Processes 

Two  ideas  are  presented  in  this  section,  the  first  concerning  the  content  of  training  and  the 
second  the  method.  Regarding  content,  the  various  types  of  knowledge  presented  in  Table  2  are 
the  results  of  research  studies  and  cognitive  task  analyses,  which  involve  formal  and  informal 
observation  of  experts’  problems-solving  performance.  The  goal  of  these  studies  and  analyses  is 
to  reveal  the  explicit  and  implicit  (tacit)  aspects  of  experts'  problem-solving  knowledge.  A 
number  of  educational  researchers  have  suggested  that  an  effective  program  for  teaching  a 
complex  cognitive  skill  like  troubleshooting  should  explicitly  teach  all  the  types  of  knowledge 
necessary  for  expert  task  performance.  (Collins,  Brown  &  Newman,  1989;  Gott,  1988).  Some  of 
the  key  types  of  troubleshooting  knowledge  in  Table  2,  such  as  strategies  and  metacognitive 
(coordination)  knowledge,  often  are  not  taught  explicitly  in  maintenance  training  (Means  &  Gott, 
1988).  Strategic  and  metacognitive  knowledge  is  not  included  exphcitly  in  TOs,  so  technicians 
will  not  pick  it  up  on  the  job  from  using  TOs. 

Regarding  methods  of  training,  a  number  of  researchers  have  suggested  that  problem-based 
learning  is  very  effective  in  teaching  complex  skills  like  troubleshooting  and  maintenance 
(Collins  et  al.,  1989;  Gott,  1988).  That  is,  students  need  to  practice  solving  problems  similar  to 
those  they  will  solve  on  the  job.  This  focus  on  realistic  problem-solving  is  used  in  Air  Force 
OJT.  Since  OJT  is  a  major  emphasis  of  this  paper,  we  will  focus  on  a  training  method  called 
cognitive  apprenticeship  training  (Collins  et  al.,  1989).  This  method  fits  well  with  the  specific 
training  techniques  used  in  Air  Force  OJT,  including  problem-based  learning.  However, 
apprenticeship-training  techniques  can  also  be  used  in  the  classroom,  so  this  approach  will  help 
in  understanding  instruction  in  the  Technical  School  and  FTD  classrooms.  In  addition,  a  key 
potential  use  of  IMIS  in  training  is  as  a  simulator,  where  students  can  practice  solving  realistic 
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maintenance  problems  (Babiarz  et  al.,  1989;  Link  et  al.,  1987).  Thus,  the  apprenticeship-training 
approach  is  useful  in  understanding  how  IMIS  can  be  used  in  training. 

As  mentioned  above,  apprenticeship  training  is  problem-based.  Whether  it  takes  place  in  a 
classroom  or  on  the  job,  much  of  the  learning  takes  place  while  students  solve  problems  similar 
to  those  they  will  face  in  the  workplace.  The  techniques  of  apprenticeship  training  include: 
explicit  modeling  by  the  teacher  of  the  skills  needed  for  expert  performance;  coaching,  in  which 
the  teacher  supports  the  students  via  hints  or  reminders  during  problem-solving;  and  fading  in 
which  the  teacher  gradually  withdraws  support  as  the  student  progress.  These  techniques  focus 
on  the  cognitive  aspects  of  learning.  In  addition,  apprenticeship  training  also  emphasizes  the 
social  context  of  learning.  For  example,  collaboration  and  learning  in  multiple  contexts  is 
encouraged.  Apprenticeship  training  has  been  found  to  be  effective  in  teaching  complex 
cognitive  skills,  including  troubleshooting  (Collins  et  al.,  1989;  Lajoie  &  Lesgold,  1989). 

3.2  Findings  from  Observations  of  IMIS 

One  of  the  main  purposes  of  IMIS  is  to  integrate  the  wide  range  of  information  needed  to 
perform  maintenance  activities  and  give  technicians  easy  access  to  this  information  through  a 
single  system.  IMIS  is  intended  to  integrate  the  following  kinds  of  information: 

•  Technical  orders 

•  Diagnostic  information 

•  Flight  data 

•  Aircraft  historical  data 

•  Supply  and  management  data 

•  Training  data 

Currently,  only  the  technical  orders  and  diagnostic  information  are  available  in  the  IMIS 
prototype. 

Figure  1  shows  the  relationships  among  the  current  IMIS  hardware  components.  In  its 
present  configuration,  the  IMIS  hardware  consists  of  the  Maintenance  Workstation,  the  Portable 
Maintenance  Aid  (PMA),  and  the  Aircraft  Interface  Panel.  The  workstation  is  used  during  the 
early  stages  of  maintenance  to  help  debrief  the  pilot,  analyze  flight  and  historical  data,  schedule 
the  maintenance  activity,  and  assemble  a  set  of  TOs  and  other  information  that  can  assist  the 
technician  during  troubleshooting  and  repair.  A  portion  of  this  information  is  downloaded  into  a 
memory  module  which  is  inserted  into  the  PMA.  The  PMA  is  a  mgged  computer  about  the  size 
of  a  commercial  laptop.  The  technician  takes  the  PMA  to  the  flightline  and  hooks  it  up  to  the 
plane  with  the  Aircraft  Interface  Panel.  During  troubleshooting  and  repair,  the  technician  has 
access,  via  the  PMA,  to  any  TOs  that  are  needed  and  to  additional  advice  from  IMIS's  Diagnostic 
Module.  The  PMA  can  also  access  information  from  the  aircraft,  such  as  built-in-test  data.  The 
technician  can  use  the  PMA  to  order  parts  and  to  issue  maintenance  status  reports  via  a  radio  link 
to  the  workstation. 
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Figure  1.  Relationships  Among  IMIS  Hardware  Components 

One  of  the  main  components  of  IMIS  that  can  be  used  in  training  is  the  Diagnostic  Module 
(DM)  (Cooke  et  al.,  1991).  This  software  module  provides  expert  troubleshooting  advice  to  the 
technician.  The  DM  uses  data  from  an  IMIS  database  called  the  CDM  (Earl  et  al.,  1990).  The 
CDM  contains  two  types  of  information.  The  first  is  information  from  TOs.  TOs  are  exphcit, 
step-by-step  procedures  for  performing  troubleshooting  and  repair  activities.  A  major  advantage 
of  IMIS  over  the  current  system  of  paper  TOs  is  that  IMIS  saves  technicians  the  trouble  of 
carrying  thousands  of  pages  of  manuals  to  the  flightline  and  searching  back  and  forth  from  one 
manual  to  another  during  troubleshooting.  However,  IMIS  provides  more  diagnostic  aid  than 
just  on-line  TOs.  The  second  type  of  information  in  the  CDM  is  data  on  common  aircraft 
symptoms,  the  faults  that  could  cause  specific  symptoms,  fault  probabilities  (i.e.,  component 
failure  rates),  and  the  expected  results  of  tests  and  replacements. i  These  data  are  used  by  the 
DM  to  provide  advice  during  troubleshooting  beyond  that  contained  in  the  TOs. 

At  the  start  of  a  troubleshooting  problem,  a  description  of  the  symptoms  exhibited  by  the 
aircraft  is  input  into  the  PMA.  The  DM  then  creates  a  recommended  list  of  tests  and  component 
replacements  that  could  help  isolate  (or  locate)  the  faulty  component.  This  list  is  rank  ordered, 
with  the  most  effective  troubleshooting  action  shown  first.  The  technician  is  free,  however,  to 
perform  any  troubleshooting  action  he  or  she  wants,  including  actions  not  on  the  recommended 
list.  After  a  test  or  replacement  is  completed,  the  information  from  this  action  is  input  into  the 
PMA  and  the  DM  calculates  a  new  list  of  troubleshooting  actions.  This  process  is  repeated  until, 
in  most  cases,  the  technician  isolates  the  fault.  Then  the  PMA  will  show  the  technician  the 
appropriate  TO  to  help  him  or  her  repair  the  fault. 

1 A  symptom  is  the  outward  manifestation  of  a  problem  with  the  aircraft.  A  fault  (e.g.,  a  broken  component)  is  the 
underlying  cause  of  a  symptom. 
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The  DM  considers  a  wide  variety  of  information  in  order  to  choose  recommended  tests  and 
replacements,  including  the  faults  commonly  associated  with  the  current  symptoms,  the  failure 
rates  of  potentially  faulty  components,  the  time  to  perform  tests  and  replacements,  and  the 
availability  of  replacement  parts.  In  order  to  assess  IMIS's  potential  as  a  training  system,  it  is 
helpful  to  compare  the  knowledge  and  processes  used  by  the  DM  with  those  used  by  human 
experts  during  troubleshooting.  As  shown  in  Table  2,  human  troubleshooters  use  mental-models, 
symptom-fault-associations,  procedures,  strategies,  and  coordination  processes. 

The  DM  uses  extensive  symptom-fault-association  and  procedures  knowledge.  The  data  in 
the  CDM  on  symptoms,  faults,  failure  probabilities,  and  test  and  replacement  results  are 
equivalent  to  symptom-fault  associations.  The  CDM  data  on  TOs  are  examples  of  procedures 
knowledge. 

In  addition,  the  DM  uses  two  troubleshooting  strategies  that  are  commonly  used  by  expert 
technicians.  The  first  of  these  is  elimination.  Recall  that  in  this  strategy,  all  components  that  are 
spanned  by  (i.e.,  lead  into)  a  troubleshooting  test  yielding  a  successful  result  (a  pass)  are 
eliminated  firom  the  set  of  potentially  faulty  components.  Figure  2  shows  a  screen  from  an  early 
version  of  IMIS  during  troubleshooting  of  a  fault  in  the  F- 16  Stores  Management  System.  The 
recommended  test  (3 1 AH2)  is  listed  in  the  upper  right  comer,  along  with  other  test  options.  The 
components  outlined  in  gray  are  the  set  of  potentially  faulty  components  diagnosed  by  IMIS, 
given  the  current  information.  The  components  shaded  in  black  are  those  that  will  be  eliminated 
if  the  recommended  test  passes.  If  the  technician  performs  this  test  and  it  passes,  then,  on  the 
next  IMIS  screen,  these  components  will  not  be  outlined  in  gray. 


STORES  MANAGEMENT  SYSTEM 
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Figure  2.  IMIS  (PMLA)  Display  During  Troubleshooting  (from  Babiarz,  et  al.,  1989) 
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The  DM  also  uses  a  version  of  half  split  (or  binary  search),  a  strategy  that  is  used  by  human 
experts.  A  typical  use  of  half  split  by  a  person  employs  topographic  mental-model  knowledge. 
For  example,  if  the  person  knows  that  the  current  set  of  potentially  faulty  components  is 
connected  in  a  chain,  then  the  half-split  strategy  would  involve  testing  a  component  in  the  middle 
of  the  chain.  This  would  ensure  that,  whether  the  test  passes  or  fails,  half  of  the  components  can 
be  eliminated.  Note  that  the  test  recommended  by  the  DM  in  Figure  2  would  reduce  the  set  of 
potentially  faulty  components  in  half,  reflecting  the  half-split  strategy.  However,  the  tests 
recommended  by  the  DM  do  not  always  conform  to  the  strict  half-split  strategy,  because  the  DM 
considers  other  information  in  addition  to  the  half-split  strategy  when  choosing  tests,  information 
such  as  component  failure  rates  and  the  time  to  perform  tests  and  replacements. 

In  terms  of  coordination  processes,  the  DM  considers  some  of  the  same  kinds  of  information 
that  a  person  would  in  planning  how  to  attack  a  troubleshooting  problem  and  deciding  what 
strategies  to  use.  This  includes  information  such  as  the  time  to  perform  tests  and  replacements, 
and  parts  availability.  However,  the  DM's  overall  coordination  processes  are  much  different  than 
humans.  The  DM  algorithm  uses  an  exhaustive-search  approach,  evaluating  every  fault  (in  its 
database)  that  could  possibly  cause  the  current  symptoms,  and  every  test  that  could  possibly  give 
information  about  the  most  likely  fault.  These  faults  and  tests  are  evaluated  via  extensive 
numerical  computations  that  are  not  used  by  human  troubleshooters. 

Another  key  difference  between  the  DM  and  human  troubleshooters  is  that  the  DM  uses  very 
little  mental-model  knowledge,  that  is,  very  little  knowledge  of  the  system  functions  and 
topography,  in  its  reasoning.  One  kind  of  mental-model  knowledge  that  is  used  by  the  DM  is 
knowledge  of  the  components  spanned  by  different  troubleshooting  tests.  This  knowledge  is 
used  by  the  elimination  and  half-split  strategies.  Other  than  this,  the  DM  relies  primarily  on 
symptom-fault  association  knowledge,  rather  than  mental-model  knowledge.  This  emphasis  is 
reflected  in  the  block  system  diagram  in  Figure  2.  This  diagram  is  based  on  "fault  connectivity 
rather  than  functional  connectivity"  (Cooke  et  al.,  1991,  p.  42).  A  diagram  showing  functional 
and  topographic  connectivity  (e.g.,  a  schematic)  would  contain  mental-model  information. 

These  similarities  and  differences  between  the  human  and  the  DM's  approach  to 
troubleshooting  will  affect  how  IMIS  can  be  used  as  an  intelligent  training  system.  For  example, 
a  key  feature  of  an  intelligent  training  system  is  its  abiUty  to  explain  its  reasoning  processes  to 
students.  A  system's  explanation  capabihties  will  be  more  effective  to  the  extent  that  its 
reasoning  processes  approximate  those  of  a  human  expert  (Anderson,  1988).  The  above  analysis 
shows  that,  although  the  DM  was  not  intended  to  simulate  human  troubleshooting  processes,  its 
knowledge  and  reasoning  processes  overlap  considerably  with  those  used  by  expert 
troubleshooters.  In  Anderson's  (1988)  terms,  the  DM  is  somewhere  between  a  black-box  expert 
that  reasons  completely  unlike  humans  and  a  glass-box  expert  that  accurately  mimics  human 
reasoning.  In  its  current  capacity  as  a  job  aid,  the  DM  can  explain  its  recommended  tests  and 
replacements  using  the  following  information  from  the  CDM:  the  expected  results  of  tests  and 
replacements  (symptom-fault  association  knowledge)  and  the  time  for  tests  and  replacements. 

It  would  be  easy  for  the  DM  to  construct  explanations  based  on  other  symptom-fault-association 
knowledge  and  troubleshooting  strategies,  and  more  difficult  (though  not  impossible)  to 
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construct  human-like  explanations  in  term  of  mental-model  knowledge  and  coordination 
processes. 

Research  by  Kieras  (1992)  suggests  that  in  order  to  reason  effectively  about  malfunctioning 
systems,  technicians  need  mental-model  information  about  the  topographic  structure  of  the 
system.  However,  the  fact  that  the  DM  does  not  use  much  mental-model  knowledge  is  not  a 
major  shortcoming  in  terms  of  its  use  as  an  instructional  system.  Very  few  computer-based 
systems  can  reason  flexibly  about  a  system  using  mental-model  knowledge.  Those  that  can,  such 
as  the  RAPIDS  system  (Towne  &  Munro,  1991),  have  not  been  implemented  for  systems  as 
complex  as  the  F-16.  In  addition,  no  existing  computer-based  systems  use  all  of  the  knowledge 
that  an  expert  troubleshooter  does.  Ways  of  compensating  for  the  DM's  de-emphasis  of  mental- 
model  knowledge  will  be  discussed  in  a  later  section. 

Another  key  feature  of  an  intelligent  training  system  is  its  ability  to  construct  a  model  of  an 
individual  student's  knowledge  and  reasoning  (VanLehn,  1988).  A  student  model  may  be 
important  if  MIS  is  to  be  used  as  part  of  an  intelligent  tutoring  system  (US)  that  tracks  students’ 
knowledge  and  performance  and  adjusts  its  instruction  to  students'  current  knowledge  state. 
IMIS's  lack  of  human-like  mental-model  and  coordination  knowledge  may  hamper  its  ability  to 
create  an  effective  student  model.  However,  some  researchers  have  argued  that  useful  USs  can 
be  developed  without  detailed  student  modeling  (Newman,  1989).  How  IMIS’s  capabilities  will 
fit  into  an  intelligent  training  system  will  be  discussed  further  in  a  later  report. 

Our  final  comment  concerning  IMIS's  capabilities  focuses  on  the  accuracy  and  usefulness  of 
its  troubleshooting  advice.  Since  MIS  uses  troubleshooting  knowledge  and  strategies  beyond 
that  in  the  TOs,  one  would  expect  that  MIS  would  provide  more  accurate  troubleshooting  advice 
and  would  help  technicians  solve  some  troubleshooting  problems  that  could  not  be  solved  using 
only  the  TOs.  However,  the  evaluations  of  MIS  conducted  so  far  have  not  provided  convincing 
evidence  that  IMIS  is  more  accurate  that  the  TOs.  For  example,  an  evaluation  conducted  in  the 
summer  of  1992  at  the  Marine  Corps  Air  Station,  Beaufort,  SC  compared  the  performance  of 
technicians  using  MTS  and  TOs.  Troubleshooting  accuracy  was  measured  by  variables  such  as 
the  number  of  failures  to  find  a  fault  and  the  number  of  unnecessary  line-replaceable-unit  (LRU) 
removals.  However,  there  were  so  few  cases  of  failure  to  find  a  fault  or  unnecessary  LRU 
removal  that  the  question  of  whether  MIS  outperformed  the  TOs  on  these  variables  could  not  be 
answered  (E.  Carlson,  personal  communication,  November  15,  1992).  The  question  of  whether 
IMIS  is  more  accurate  than  the  TOs  is  important  in  evaluating  MIS's  potential  effectiveness  as  a 
training  system,  as  will  be  discussed  further  later  in  this  report. 

3.3  Findings  Concerning  Other  Automated  Systems 

In  this  section  we  describe  the  characteristics  of  other  automated  Air  Force  systems  that 
might  have  an  impact  on,  or  enhance,  those  capabihties  of  IMIS  which  would  be  used  to  support 
Air  Force  maintenance  training.  Later  in  this  report  we  will  develop  hypotheses  about  how  MIS 
might  interact  with  these  systems. 
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3.3.1  Base  Training  System  (BTS) 


This  system  is  expected  to  exist  at  all  operational  bases  in  support  of  the  OJT  environment. 
Currently  it  consists  of  three  training  management  components  each  of  which  is  described  below. 

Training  Requirements  Management.  This  component  provides  the  following: 

•  Identification  of  all  pertinent  performance  and  training  requirements  that  are  managed 
at  Air  Force  job  sites; 

•  Identification  of  references  for  performance  and  training  requirements; 

•  Prioritization  of  performance  and  training  requirements; 

•  Assessment  of  personnel  qualifications  to  determine  suitability  for  assignment  to 
specified  duty  positions;  and 

•  Development  of  behavioral  objectives. 

Training  Progress  Management.  This  component  provides  the  capability  to  perform  the 
following  functions: 

•  Generate  and  maintain  on-line  training  records; 

•  Schedule  training  and  evaluation  events;  and 

•  Generate  standard  and  ad-hoc  training  effectiveness  reports. 

Resource  Inventories  Management.  This  component  provides  the  capability  to  identify  and 
maintain  an  inventory  (list)  of  resources  required  for  performance,  training,  and  evaluation  of 
specialty  tasks.  Specified  are  resources  required  for  each  task,  subtask,  behavioral  objective,  and 
event  within  a  specialty.  (Vigue  &  Young,  1991,  p.  5) 

3.3.2  Advanced  Training  System  (ATS) 

This  system  will  exist  at  the  five  Technical  Training  Centers  for  the  purpose  of  design, 
development,  delivery,  evaluation,  and  course  management  for  various  training  activities.  It  will 
include  some  of  the  following  capabilities: 

•  Design  tools  for  lesson  plans,  sequences,  flow  charts,  and  storyboards; 

•  Automate  course  control  documentation  with  cross-referencing  to  requirements  and 
resources; 

•  Provide  student  remediation  and  practice; 

•  Monitor  student  performance  and  status  monitoring; 

•  Review  training  data; 

•  Provide  on-line  coordination,  review,  and  editing  of  training  documents;  and 

•  Schedule  students,  instructors,  equipment,  and  facilities. 

Currently  this  system  will  communicate  with  the  Occupational  Measurement  Squadron 
system  (for  task  analysis  data),  the  Extension  Course  Institute  (for  Career  Development  Course 
materials),  and  the  Air  Force  Training  Management  System  (for  class  schedule  and  student 
accounting  information). 
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3.3.3  Automated  Instructional  Design  Advisor  (AIDA) 

This  system  was  designed  to  be  an  automated  and  integrated  collection  of  tools  to  assist  in 
the  design,  development,  and  delivery  of  instruction.  Its  goal  is  to  provide  intelligent  and 
automated  assistance  throughout  all  phases  of  instructional  development,  particularly  computer- 
based  instructional  development,  for  general  maintenance  training.  Thus,  AIDA  contains 
instructional  expertise  which  allows  a  subject-matter  expert  to  directly  author  instruction  without 
needing  the  advice  or  assistance  of  an  expert  instructional  designer. 

3.3.4  Core  Automated  Maintenance  System  (CAMS) 

This  system  exists  in  active  duty  and  reserve  units  for  the  purpose  of  automating  and 
integrating  data  concerning  maintenance  activities.  Data  can  be  recorded  and  stored  for  such 
maintenance  functions  as  status  and  inventory,  operational  events,  automated  test  equipment 
reporting  system,  equipment  transfer  procedures,  personnel  transfer  procedures,  and  training 
management,  to  name  a  few.  It  also  interfaces  with  the  Standard  Base  Supply  System  (SBSS). 

The  training  management  portion  of  this  system  allows  personnel  to  record  and  maintain  data 
on  ancillary  training  and  OJT.  At  the  reserve  units  training  status  information  for  personnel  can 
be  recorded  and  viewed  at  the  course  level,  at  the  work  center  level,  or  at  the  individual  level. 
For  instance,  a  typical  printout  for  an  individual  would  list  all  relevant  courses  of  instruction  for 
his  or  her  specialty  with  the  status  for  each  course  being  given  as  completed,  overdue,  or  due  for 
annual  update.  In  the  active  duty  units  status  information  at  the  task  level  would  be  given  as 
well  for  individuals  and  for  work  centers. 

In  order  to  log-on  to  the  system,  personnel  (from  technicians  to  supervisors  to  trainers)  must 
have  an  i.d.  number  and  password.  Personnel  with  this  information  may  record  relevant  data 
(such  as  aircraft  repair  information)  or  view  relevant  information  (such  as  personnel  status  for  a 
workcenter).  The  computer  recognizes  each  identify  number  (ID)  and,  based  on  pre-determined 
access  information  for  various  personnel,  allows  an  individual  to  view  only  documentation  that 
would  be  relevant  to  his  or  her  duties.  Certain  documentation  must  be  requested  through  the 
CAMS  operations  personnel. 

3.4  Findings  Concerning  the  Maintenance  Process 

In  this  section  we  describe  the  F-16  maintenance  process.  Although  the  main  focus  of  this 
report  is  the  maintenance  training  process,  coverage  of  the  maintenance  process  itself  is 
necessary  for  the  following  reasons.  As  mentioned  before,  one  must  understand  the  nature  of  a 
task  before  designing  training  for  that  task.  Since,  in  a  later  phase  of  this  project,  we  will 
develop  a  computer-based  demonstration  of  how  IMIS  can  be  used  in  F-16  maintenance  training, 
we  need  to  first  understand  the  maintenance  process.  An  understanding  of  this  process  leads  to  a 
number  of  specific  recommendations  for  how  IMIS  can  be  used  in  training.  These  are  described 
later  in  the  report. 
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Table  3 

Demographic  Information  for  Kellv  AFB  Maintenance  Supervisors  rN=5) 


Extent  of  Experience 
(answered  in  number  of  years) 

Mean 

Standard  Deviation 

How  long  in... 

Air  Force/Air  National  Guard 

22.2 

11.1 

Maintenance  Career  Field 

19.4 

7.8 

Current  Specialty 

19.4 

7.8 

Current  Work  Center 

13 

8.8 

Maintenance  Supervisor 

14 

1 

OJT  Trainer 

14.3 

8.2 

Questionnaire  data  for  25  Air  Force  personnel  and  5  Air  National  Guard  (ANG)  personnel, 
and  observational  data,  were  used  for  this  analysis.  Demographic  information  for  these 
personnel  is  shown  in  Tables  3, 4,  5,  and  6.  These  tables  are  broken  down  by  base  (Kelly  AFB, 
Hill  AFB  and  Lowry  AFB)  and,  when  necessary,  by  category  of  personnel.  Demographic 
information  applies  to  this  section  as  well  as  the  following  section  on  the  training  process. 


Table  4 

Demographic  Information  for  Lowrv  AFB  Technical  School  Instructors  and  Course  Developer 
for  Avionics  (452X2)*  (N=5) 


Extent  of  Experience  (answered 
in  number  of  years) 

Mean 

Standard  Deviation 

How  long  in... 

Air  Force 

10.7 

Maintenance  Career  Field 

10.45 

5.01 

Current  Specialty 

7.7 

6.4 

Tech  School  Instructor,  Course 
Supervisor,  etc. 

3.6 

1.5 

*One  instructor  interviewed  was  a  course  supervisor  as  well.  (The  same  questionnaire  format  was 
used  for  all  five  personnel) 


Our  sample  size  is  relatively  small  due  to  various  constraints  in  collecting  data.  However,  we 
feel  our  results  are  a  fair  representation  of  what  actually  occurs  in  Air  Force  maintenance  and 
maintenance  training  environments.  This  is  not  to  say  that  our  data  necessarily  reflect  specific 
occurrences  in  these  environments,  rather  a  general  representation.  We  base  this  confidence  not 
only  on  the  data,  but  on  literature  reviews  of  these  environments  and  upon  the  advice  of  subject 
matter  experts  other  than  those  interviewed. 
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Table  5 

Demographic  Information  for  Hill  AFB  Technicians  (N=6) 


Extent  of  Experience 
(answered  in  number 
of  years) 

Avionics 

(N=2) 

Mean  Std.  Dev. 

Environm. 

(N=l) 

Pneudraulics 

(N=3) 

Mean  Std.  Dev. 

How  long  in... 

Air  Force 

7.5 

6.4 

1.5 

12.7 

4.9 

Maintenance  Career 
Field 

7 

7.1 

1.5 

12.7 

4.9 

musmmmm 

4.5 

3.5 

1.5 

10.8 

7.5 

Current  Work  Center  or 
Detachment 

2 

0 

1.5 

1.7 

.29 

3.4.1  The  F-16  Maintenance  Process 

Aircraft  maintenance  activities  fall  into  two  categories:  scheduled  and  unscheduled. 
Scheduled  maintenance  occurs  when  aircraft  are  temporarily  taken  out  of  service  for  the  purpose 
of  performing  routine  preventative  maintenance.  Activities  include  removing  and  installing 
components,  aircraft  overhauls,  and  operational  checkouts  (which  involve  checking  whether  a 
subsystem  is  working  correctly). 

Unscheduled  maintenance  occurs  as  a  result  of  aircraft  malfunctions  requiring  immediate 
repair.  It  consists  of  the  troubleshooting  and  repair  work  necessary  to  get  malfunctioning  aircraft 
back  in  service.  This  usually  involves  three  processes:  fault  detection,  fault  isolation  (locating 
the  cause  of  a  problem),  and  fault  recovery  (repair).  We  use  troubleshooting  as  a  general  term 
encompassing  all  three  of  these  processes.  Two  key  factors  affect  the  way  in  which  F-16 
unscheduled  maintenance  is  performed  in  the  Air  Force.  The  first  of  these  is  system  complexity. 
T  .ikft  any  modem  jet  aircraft,  the  F-16  is  a  very  complex  system,  with  subsystems  for  propulsion, 
avionics,  environmental  control,  communications,  and  other  functions.  A  large  array  of 
equipment  is  tightly  fit  into  a  relatively  small  fuselage.  Often,  in  order  to  test  or  replace  a 
component,  a  technician  must  remove  other  components  that  block  access  to  it. 

The  second  factor  affecting  F-16  unscheduled  maintenance  has  to  do  with  the  time 
limitations  for  performing  the  maintenance.  On  the  flightline,  where  much  of  the  unscheduled 
maintenance  occurs,  planes  must  often  be  "turned"  -  that  is,  repaired  and  readied  for  flight  -  in  a 
matter  of  a  few  hours  (in  combat,  even  less). 
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Table  6 

Demographic  Information  for  Supervisors  and  FTP  Instructors  in  each  AFSC  (N=14')  and  the  Maintenance  Training  Manager  (N=l')  at 
Hill  AFB 
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The  twin  factors  of  systems  complexity  and  time  constraints  have  led  the  Air  Force  to 
develop  a  system  of  LRUs  and  standardized  troubleshooting  and  repair  procedures  (TOs)  in  order 
to  simplify  and  speed  up  maintenance  procedures.  An  LRU  is  a  component  or  set  of  components 
that  can  be  removed  from  the  aircraft  as  a  unit  and  sent  to  a  repair  shop  for  further 
troubleshooting  and  possibly  repair.  Using  LRUs  cuts  down  the  amount  of  troubleshooting 
maintenance  technicians  have  to  do  on  the  flightline.  TOs  provide  a  set  of  procedures  that 
frequently  allow  a  maintenance  technician  to  find  the  faulty  LRU  without  time  consuming 
problem  solving. 

3.4.2  The  Maintenance  Process  at  Hill  Air  Force  Base 

Two  groups  of  people  are  involved  in  maintenance  activities  at  Hill.  The  first  group  consists 
of  the  maintenance  technicians  and  supervisors  who  perform  the  initial  fault  isolation  and 
recovery  on  the  flighthne.  Some  of  the  main  resources  used  by  maintenance  technicians  are  TOs, 
which  include  job  guides  (detailed  procedures  for  operational  checkouts  and  for  removing  and 
replacing  components)  and  fault  isolation  charts  (decision  trees  to  aid  in  fault  isolation). 

In  performing  their  jobs,  maintenance  technicians  often  isolate  a  fault  to  an  LRU  and  then 
remove  and  replace  the  LRU.  The  faulty  LRU  is  then  sent  to  the  "back  shop."  There,  a  second 
group  of  technicians  use  automatic  test  equipment  to  diagnose  the  fault  within  the  LRU  and 
repair  it,  if  possible. 

As  mentioned  above,  the  maintenance  process  (at  least  for  unscheduled  maintenance) 
consists  of  fault  detection,  fault  isolation  and  fault  recoveiy.  Sometimes  fault  detection  occurs 
when  pilots  notice  problems  in  a  plane's  operation  during  flight.  Other  times,  faults  are  detected 
by  maintenance  technicians  or  on-board  computers  during  pre-  or  post-flight  operational 
checkouts.  The  on-board  computers  that  test  for  faults  are  called  built-in-test  equipment. 

The  first  step  in  fault  isolation  is  to  describe  the  problem  (i.e.,  the  symptoms).  A 
maintenance  technician  initially  receives  a  work  order  containing  a  brief  description  of  the 
problem,  which  is  obtained  from  a  pilot  debriefing.  Technicians  may  question  the  pilot  further 
about  the  specifics  of  when  and  how  the  problem  occurred.  The  second  step  is  to  isolate  the  fault 
using  a  fault-isolation  (FI)  chart  and  other  TOs.  The  FI  chart  provides  a  decision  tree  telling  the 
technician  what  tests  and  component  replacements  to  make  and  how  to  interpret  the  results  of 
these  tests  and  replacements.  To  perform  a  component  replacement  required  by  an  FI  chart,  a 
technician  often  needs  to  use  a  job-guide  TO. 

The  need  to  switch  among  several  TOs  while  performing  a  single  task  causes  a  number  of 
problems.  First,  the  TOs  are  bulky  and  technicians  often  have  to  carry  several  TO  manuals 
comprising  thousands  of  pages.  Second,  and  more  importantly,  information  in  TOs  is  at  times 
hard  to  locate  and  technicians  can  sometimes  get  lost  while  using  them.  For  example,  Tenney 
and  Kurland  (1988)  report  a  case  of  an  expert  radar  technician  turning  to  the  wrong  procedure  in 
a  manual  and  having  a  hard  time  recovering  from  this  error.  Another  significant  problem  with 
Fl-chart  TOs  is  that  they  fail  to  isolate  some  faults.  A  final  problem  complicating 
troubleshooting  is  that,  although  efforts  are  made  to  update  TOs  regularly  to  correspond  to 
aircraft  changes,  sometimes  the  TOs  are  not  quite  up  to  date.  When  a  technician  gets  lost  in 
following  a  TO,  or  the  TO  does  not  isolate  a  fault,  he  or  she  must  use  problem-solving  skills  to 
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recover  from  this  situation.  These  skills  include  using  some  of  the  kinds  of  troubleshooting 
knowledge  described  above,  such  as  mental-model  knowledge  and  troubleshooting  strategies. 

We  collected  data  on  the  extent  to  which  maintenance  technicians  have  to  go  beyond  the  TOs 
in  flightline  troubleshooting.  One  question  asked  supervisors  and  maintenance  technicians  to 
estimate  the  percentage  of  technical-order  faults  and  problem-solving  faults  encountered  by 
technicians..  Problem-solving  faults  were  defined  as  those  where  "the  technical  orders  do  not 
provide  enough  information  to  diagnose  or  repair  the  fault.  Here  the  technician  must  make  major 
decisions  about  what  tests  or  repairs  to  make  and/or  how  to  sequence  these  tests  and  repairs."  All 
other  faults  were  defined  as  technical-order  faults.  The  mean  percentage  of  problem-solving 
faults  estimated  by  the  20  technicians  and  supervisors  interviewed  was  30%  (median  =  22.5%). 
Thus,  technicians  must  use  information  and  knowledge  beyond  that  in  the  TOs  for  30%  of  the 
problems  they  face. 

What  troubleshooting  methods  do  technicians  use  when  they  must  go  beyond  the  TO?  We 
listed  five  methods  of  troubleshooting  and  asked  technicians  and  supervisors  to  estimate  the 
percentage  of  troubleshooting  tasks  on  which  the  average  technician  would  use  each  method. 
The  methods  and  mean  percentage  estimates  are  shown  in  Table  7.  These  data  suggest  that 
technicians  often  use  methods  other  than  following  TOs,  including  some  methods  that  involve 
problem-solving  skills,  such  as  using  analogies  and  troubleshooting  strategies. 


Table  7 

Troubleshooting  Methods  Used 


Method 

Mean  Percentage 
of  use 

Follow  technical  orders 

64% 

Recall  how  he/she  solved  the  same  problem  before  and  use  this  procedure 

56% 

Recall  how  he/she  solved  a  similar  problem  before  and  use  this  as  an 
analogy  or  example 

51% 

Ask  co-workers  for  help 

36% 

Use  troubleshooting  strategies  (such  as  working  backwards  from  the 
symptom,  eliminating  components  known  to  be  working,  and  half  split) 

46% 

3.4.3  Summary  of  the  Maintenance  Process 

The  complexity  of  the  F-16  and  the  time  pressures  inherent  in  flightline  maintenance  led  the 
Air  Force  to  institute  a  standardized  maintenance  system  based  on  TOs  and  LRUs.  Most  of  the 
time,  this  system  achieves  its  goal  of  simplifying  and  speeding  up  the  maintenance  process. 
However,  it  is  not  always  successful.  Sometimes,  technicians  get  lost  or  misinterpret  the  TO,  or 
the  TO  cannot  isolate  a  fault.  Our  data  show  that  for  about  30%  of  unscheduled  maintenance 
problems,  technicians  must  go  beyond  the  TOs  and  use  other  approaches  such  as  troubleshooting 
strategies  and  analogies  to  similar  problems.  This  30%  figure  is  of  practical  significance, 
especially  since  the  difficult  problems  that  require  more  than  TO  knowledge  probably  take  up 
much  more  than  30%  of  unscheduled  maintenance  time. 
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The  pilot  data  we  collected  at  Kelly  AFB  shows  a  pattern  similar  to  the  Hill  AFB  data.  Four 
maintenance  supervisors  at  Kelly  estimated  that,  on  average,  technicians  must  go  beyond  the  TOs 
on  47%  of  unscheduled  maintenance  problems,  and  that  technicians  use  troubleshooting  methods 
other  than  TOs  (i.e.,  the  methods  in  Table  7)  from  20  to  40%  of  the  time. 

A  number  of  maintenance  personnel  at  Hill  and  Kelly  AFBs  described  a  problem  with  the 
current  reliance  on  TOs  and  LRUs.  The  TO/LRU  system  has  led  to  a  two-tiered  system  of 
maintenance  technicians,  less  experienced  LRU  swappers  who  must  rely  on  TOs,  and  expert 
technicians  who  can  go  beyond  the  TO  procedures  to  diagnose  and  repair  the  more  difficult 
problems.  Our  earlier  analysis  of  the  knowledge  required  for  expert  troubleshooting  (see  Table 
2)  suggests  that  the  less  experienced  technicians  primarily  are  using  procedures  knowledge,  while 
the  experts  are  using  a  wide  range  of  troubleshooting  knowledge,  including  procedures, 
strategies,  coordination  processes,  mental  models,  and  symptom-fault  associations. 

In  our  description  of  the  maintenance  process  at  Hill  AFB,  we  have  emphasized 
troubleshooting  tasks,  and,  in  particular,  the  problem-solving  aspects  of  troubleshooting.  We 
chose  this  emphasis  because  of  the  difficulty  and  importance  of  these  tasks  (as  discussed  here 
and  in  the  literature  review  section).  However,  we  wish  to  re-iterate  that  maintenance  activity 
also  includes  many  non-troubleshooting  tasks.  Many  of  these  tasks  involve  following  pre- 
leamed  procedures  (e.g.,  safety  and  administrative  procedures,  and  procedures  for  using  tools  and 
equipment).  Also,  as  our  data  show,  many  troubleshooting  tasks  consist  of  following  procedures. 

3.4.4  Conclusions  Concerning  the  Maintenance  Process. 

One  advantage  of  the  TO/LRU  system  is  that  it  lessens  the  need  for  more  extensive  training 
of  maintenance  personnel.  As  long  as  a  small  group  of  expert  technicians  is  available  to  handle 
the  more  difficult  troubleshooting  problems,  the  system  works  well  with  a  large  group  of  less 
experienced  technicians  which  must  rely  primarily  on  TOs. 

However,  the  current  system  is  vulnerable  to  the  loss  of  key  maintenance  personnel.  It  also  is 
at  odds  with  the  goals  of  the  Rivet  Workforce  program,  which  emphasizes  training  generalists. 
To  be  effective  generalists,  technicians  need  more  than  low-level  knowledge  of  procedures. 
They  need  mental  models,  strategies,  and  metacognitive  knowledge  that  will  allow  them  to  work 
effectively  on  a  wide  range  of  systems  and  aircraft. 

This  analysis  suggests  that  the  Air  Force  has  a  pressing  need  to  train  more  maintenance 
personnel  who  can  use  the  multiple  kinds  of  knowledge  required  of  expert  troubleshooters.  This 
need  raises  a  key  question  for  how  IMIS  is  to  be  used  in  training.  Can  IMIS  help  teach 
technicians  a  broad  repertoire  of  troubleshooting  skills  that  will  allow  them  to  become  problem 
solvers  instead  of  LRU  swappers?  On  the  other  hand,  since  many  maintenance  tasks  involve 
simply  following  procedures,  the  question  also  arises  as  to  whether  IMIS  can  train  these 
procedures.  Our  preliminary  answers  to  these  questions  will  be  given  later  in  this  report. 

3.4.5  Findings  Concerning  Task  Characteristics 

The  section  following  this  one  focuses  on  the  Air  Force  maintenance  training  environment. 
Before  discussing  the  training  environment,  we  will  first  consider  the  question  of  which 
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maintenance  tasks  would  be  most  amenable  to  training.  To  get  at  this  question,  one  section  of 
the  questionnaire  (the  Task  Characteristics  section,  in  Appendix  A)  asked  respondents  to  indicate 
yes  or  no  to  33  questions  regarding  task  performance.  As  with  other  questions,  these  responses 
were  given  for  each  of  the  reference  tasks  in  Table  1.  As  mentioned  earlier,  these  questions  were 
generated  from  research  on  the  topic  of  tradeoffs  between  training  and  job  aiding.  The  answers 
to  our  questions  would  be  used  to  determine  whether  emphasis  should  be  put  on  training  or  job 
aiding  for  each  reference  task. 

A  key  was  developed  whereby  each  yes  or  no  response  was  transformed  to  a  training  or  job 
aid  tally,  with  a  +1  tally  indicating  a  response  favoring  training,  and  a  -1  indicating  a  response 
favoring  job  aiding.  Responses  were  totaled  for  each  respondent  and  each  task.  These  totals 
could  theoretically  range  from  -33  to  +33.  The  sign  of  the  total  indicated  the  respondent's 
preference  for  training  (positive)  or  a  job  aid  (negative)  for  a  particular  task.  The  magnitude  of 
the  total  indicated  the  degree  of  the  respondent's  preference  for  training  or  job  aiding  for  the  task. 

Results  revealed  much  variation  in  responses  across  personnel  in  a  single  AFS  and  for  a 
single  task.  However,  when  comparing  the  various  tasks  within  an  AFS  or  between  AFSs  there 
was  no  strong  indication  of  whether  the  performance  of  any  of  these  tasks  would  benefit  more 
from  the  use  of  a  job  aid  than  from  training.  The  means  for  all  tasks  were  close  to  0,  ranging 
from  -2  to  3. 

3.5  Findings  Concerning  the  Maintenance  Training  Process 

3.5.1  Air  National  Guard  Maintenance  Training 

As  mentioned  earlier,  the  interviews  at  Kelly  AFB  were  a  pilot  testing  phase  for  the 
measurement  instruments  (questionnaires).  The  questionnaires  were  modified  based  on  feedback 
from  the  interviewees.  Revised  versions  were  then  used  at  Hill  AFB  and  Lowry  AFB. 
Interviews  that  were  conducted  included  personnel  who  were  highly  skilled  in  their  AFSC,  and 
all  five  were  supervisors.  Less-experienced  maintenance  technicians  were  not  available. 

During  this  phase  certain  training  issues  were  expressed.  First,  there  is  currently  no  training 
available  to  fill  gaps  between  information  contained  in  the  TOs  and  problems  encountered  while 
actually  troubleshooting.  If  a  technician  runs  into  a  troubleshooting  problem  not  covered  in  the 
TO,  most  often  he  or  she  has  to  either  consult  with  expert  technicians  or  attempt  to  solve  the 
problem  alone  by  using  problem-solving  skills  and  past  experience.  Second,  training  at  the  guard 
unit  at  Kelly  AFB  is  strictly  OJT.  Most  technicians  there  fall  into  three  categories:  1)  experts, 
who  have  extensive  prior  experience  and  Technical  School  training,  2)  weekend  journeymen 
maintaining  their  skills,  and  3)  novices  with  Uttle  or  no  prior  experience  who  need  extensive 
OJT.  Third,  because  the  guard  unit  is  limited  in  the  number  of  aircraft  it  maintains,  training  only 
takes  place  when  there  are  breakdowns  and  there  are  enough  personnel  and  time  to  train  a 
technician  in  the  skills  that  are  needed.  Meeting  all  these  conditions  is  very  difficult. 

3.5.2  Air  Force  Maintenance  Training 

Air  Force  technical  training  is  a  standard  and  structured  path  for  most  maintenance 
specialties.  After  basic  training,  all  Air  Force  recraits  assigned  as  maintenance  technicians  go 
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through  Technical  School.  Then,  they  are  assigned  to  a  base  where  they  receive  OJT  as  well  as 
classroom  instruction  from  a  Field-Training  Detachment  (FTD)  or  Logistics  Support  Training 
(LST).  We  collected  data  on  each  of  these  training  environments,  and  will  describe  our  findings 
for  each  environment  in  turn,  following  the  sequence  in  which  trainees  usually  encounter  them. 

Technical  School 


Our  data  on  Technical  School  training  come  from  interviews  conducted  at  Lowry  AFB, 
which  focuses  on  the  Avionics  maintenance  specialty.  Most  trainees  come  to  this  environment 
having  very  limited  or  no  knowledge  of  the  skills  needed  for  their  future  Air  Force  jobs. 
Therefore,  pre-instructional  assessment  is  not  conducted,  and  instruction  is  pre-determined  based 
on  established  course  objectives.  These  objectives  and  corresponding  blocks  of  instruction  are 
based  on  what  is  required  by  the  particular  AFS  (i.e.,  the  types  of  skills  and  knowledge  required 
to  successfully  maintain  relevant  aircraft)  and  developed  into  a  complete  course.  Each  course 
has  course  control  documentation  including  a  Plan  of  Instruction  (POI),  which  details  the  steps 
that  should  be  followed  for  proper  instruction  of  all  course  objectives.  The  POI  iS  divided  into 
sections  for  each  objective.  Each  section  lists  the  pre-determined  procedures  to  be  followed  and 
includes  personalized  notes  from  the  instructors  that  provide  more  guidance  for  carrying  out  a 
particular  lesson.  Workbooks  or  study  guides  are  normally  available  for  students.  These  include 
information  to  be  discussed  in  class,  additional  explanatory  information,  exercises,  and  quizzes 
for  each  unit  of  instruction. 

Trainees  initially  receive  eight  or  more  weeks  of  instruction  in  general  electronics  theory. 
Following  basic  electronics  training,  information  specific  to  Avionics  is  taught  in  a  twelve  week 
course.  This  course  begins  with  two  blocks  of  instruction  on  general  maintenance  procedures 
(one  being  an  introduction  to  maintenance),  followed  by  a  block  concerning  the  use  of  TOs,  and 
then  several  blocks  on  more  specific  information  for  various  subsystems. 

In  terms  of  content,  the  Avionics  course  focuses  on  teaching  functional  information  rather 
than  theory.  We  asked  two  questions  to  assess  the  content  taught  in  this  domain.  The  first 
question  was  relevant  to  a  wide  range  of  maintenance  tasks  (specifically,  the  reference  tasks  in 
Table  1).  For  each  reference  task,  respondents  were  to  indicate  which  of  the  content  areas  in 
Table  8  were  taught.^  As  the  table  shows.  Technical  School  instructors  most  frequently  teach 
system/subsystem  knowledge.  Subtasks  and  a  variety  of  procedures  are  taught  less  often. 


2  For  Tables  8  through  11,  percentages  indicate  the  proportion  of  times  an  answer  was  chosen.  Percentages  were 
obtained  by  dividing  the  actual  number  of  tallies  for  an  answer  by  the  possible  number  of  times  an  answer  could 
have  been  chosen  by  all  personnel  across  all  three  tasks. 
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Table  8 

Content  Taught  Concerning  General  Maintenance  Activities 


Content  Area 

Technical 

School 

FTD 

OJT 

System/subsystem  Knowledge 

100% 

91% 

81% 

Administrative  Procedures 

50% 

32% 

59% 

Subtasks 

50% 

27% 

91% 

Equipment/Tool  Familiarization 

42% 

82% 

78% 

Extenuating  Flightline  Procedures 

33% 

36% 

55% 

Other 

50% 

0 

0 

Our  second  question  concerning  course  content  was  relevant  to  troubleshooting  tasks  in 
particular.  It  asked  for  instructors  to  estimate  how  much  coverage  they  gave  to  the  various  kinds 
of  troubleshooting  knowledge  presented  in  Table  2  (with  1  indicating  no  coverage,  3  moderate 
coverage,  and  5  extensive  coverage).  The  results  from  this  question  are  shown  in  Table  9. 
These  data  suggest  that  Technical  School  instmctors  are  aware  of  the  various  types  of  knowledge 
necessary  for  expert  troubleshooting,  and  try  to  give  all  of  these  at  least  moderate  coverage  in 
their  classes.  The  most  extensive  coverage  was  given  to  diagnosis  and  repair  procedures  and 
mental-model  knowledge. 


Table  9 

Content  Taught  Concerning  Troubleshooting  11  =  no  coverage,  3  =  moderate  coverage,  5  - 
extensive  coverage) 


Content  Area 

Technical 

School 

FTD 

OJT 

Mental  Model-Device 

4.2 

4.7 

4.3 

Mental  Model-Topographic 

3.8 

3.3 

Symptom-Fault  Associations 

2.8 

4.3 

3.4 

Procedures  for  Diagnosis  &  Repair 

4.2 

3.7 

4.1 

Test  Procedures 

3.0 

4.0 

3.2 

Troubleshooting  Strategies 

3.6 

3.2 

Coordination  Processes 

2.8 

3.7 

3.1 

A  variety  of  teaching  methods  are  used  in  the  Avionics  course.  As  shown  in  Table  10, 
instructors  report  that  the  lecture  approach  (formal  training)  is  used  most  often.  The  other 
methods,  most  of  which  correspond  to  the  apprenticeship-training  techniques  of  modeling 
(demonstration),  coaching  (assistance),  and  independent  practice,  are  used  less  often  in  Technical 
School. 
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Table  10 

Maintenance  Training  Methods 


Type  of  Method 

Technical 

School 

FTD 

OJT 

Formal  Training 

100% 

100% 

52% 

Demonstration  and  Practice 

75% 

95% 

79% 

Oversight  and  Assistance 

25% 

45% 

78% 

Independent  Work  With  Checks 

25% 

27% 

45% 

Computer  Based  Training 

0 

5% 

10% 

Career  Development  Course 

0 

0 

5% 

One  specific  training  method  --  allowing  students  to  practice  maintenance  activities  on 
simulators  or  actual  aircraft  —  is  of  special  importance  because  it  allows  training  of  the  problem¬ 
solving  aspects  of  maintenance.  The  Avionics  course  uses  a  classroom  simulator,  which  is  an 
integrated  suite  consisting  of  a  mock-up  of  the  cockpit,  a  flatboard  panel  (where  students  practice 
test  procedures  such  as  measuring  resistances),  and  an  interactive  video  display  (where  students 
simulate  replacing  parts).  To  use  the  simulator,  the  instructor  specifies  a  fault  to  the  computer 
and  then  (acting  as  the  pilot)  debriefs  the  students  on  the  general  problem  (e.g.,  "There's  a 
problem  with  the  head-up  display.")  The  students,  who  usually  work  in  pairs,  must  then  use  their 
TOs  and  the  simulator  to  troubleshoot  the  problem.  The  instructor  sometimes  assists  the  students 
if  they  are  pursuing  an  unproductive  path  in  their  troubleshooting.  After  the  students  have  found 
the  fault,  the  instructor  sometimes  has  them  review  the  tests  and  replacements  they  made  using 
the  system  schematic,  thus  allowing  them  to  connect  mental-model  and  procedures  knowledge. 

In  the  Avionics  specialty,  at  least  one  aircraft  is  dedicated  completely  for  use  in  training.  To 
use  the  aircraft,  laboratory  time  must  be  scheduled  in  advance  in  order  to  meet  the  needs  of  all 
classes  requiring  training  of  this  type.  Again,  students  usually  work  in  pairs;  however,  the 
instructor  is  much  more  involved  in  the  learning  process.  The  instructor  demonstrates  the 
functions  of  different  parts  of  the  aircraft,  how  to  perform  operational  checks,  and  how  to 
interpret  test  results.  The  student  then  practices  the  skills  just  demonstrated,  with  frequent 
feedback  (or  coaching)  from  the  instructor 

Although  the  above  description  suggests  that  using  simulators  and  aircraft  are  particularly 
valuable  training  methods,  certain  problems  limit  their  effectiveness.  The  first  problem  is  that 
the  amount  of  time  each  student  uses  the  simulator  or  aircraft  is  extremely  limited.  Before 
trainees  get  hands-on  experience  with  the  simulator,  they  must  receive  instruction  in  how  to  use 
it.  Also,  only  one  or  two  individuals  may  use  the  equipment  at  a  time,  as  with  the  aircraft.  So 
each  individual  student  actually  receives  less  than  21  hours  experience  with  the  simulator  (4%  of 
class  time)  during  a  12  week  course,  and  even  less  with  the  aircraft  (a  total  of  4  days  for  the 
entire  class).  The  second  problem  is  that  all  the  faults  that  can  be  simulated  (using  the  current 
simulator)  are  common  ones  that  can  be  solved  by  routine  use  of  the  TOs.  Recall  that  30%  of 
flightline  faults  require  technicians  to  go  beyond  the  TOs.  Thus,  students  do  not  get  to  use  the 
simulator  to  practice  solving  these  more  difficult  faults  that  require  real  problem  solving. 
Instead,  they  use  the  simulator  only  to  practice  following  the  TOs,  which  may  be  a  necessity 
given  the  limited  time  available. 
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Table  11 

Methods  of  Evaluating  Training  Performance 


Type  of  Evaluation 

Technical 

School 

FTD 

OJT 

Checklists 

92% 

86% 

43% 

Written  Tests 

100% 

59% 

26% 

Part-task  Trainer  or  Mock-Up 

75% 

27% 

29% 

Actual  Equipment  Performance 

8% 

100% 

84% 

As  Table  1 1  indicates,  Technical  School  students  are  most  frequently  evaluated  by  written 
tests  and  through  use  of  checklists  and  part-task  trainers  or  mock-ups  (i.e.,  simulators).  Progress 
Checks  are  used  throughout  the  course  of  instruction,  while  written  tests  are  used  at  the  end  of 
each  block  of  instruction  (Comprehensive  Block  Tests).  The  instmctors  follow  a  mastery- 
learning  approach.  If  a  student  fails  a  written  test,  he  or  she  works  individually  with  the 
instructor  to  go  over  the  material  that  should  have  been  mastered.  Students  almost  always  pass 
the  test  on  the  second  try. 

Performance  is  typically  certified/decertified  by  the  instructor  noting  on  AF  Form  156 
(Knowledge  Progress  Check)  whether  an  individual  has  passed  or  failed  relevant  units  of 
instruction.  This  form  is  completed  and  maintained  by  the  instructor.  The  complete 
recordkeeping  process  typically  includes  the  instmctor  recording  student  progress  on  AF  Form 
797,  and  students'  final  grades  for  each  block  of  instraction  on  AF  Form  156. 

Field  Training  Detachment 

We  collected  data  on  FTD  classes  for  the  Avionics,  Electrical  and  Engineering,  and 
Pneudraulics  AFSs  from  the  FTD  school  at  Hill  AFB.  Most  technicians  are  assigned  to  FTD 
classes  as  soon  as  they  arrive  at  Hill  AFB.  Most  FTD  students  fall  into  one  of  two  categories:  1) 
new  students  from  Technical  School,  or  2)  those  who  have  been  Riveted.  Despite  the  fact  that 
students  have  prior  technical  training,  little  pre-instmctional  assessment  is  done  (except  for  the 
Pneudraulics  AFS),  and  instruction  is  pre-determined  based  on  the  course  objectives  and 
procedures  outlined  in  the  POIs. 

The  duration  of  the  FTD  courses  are:  Avionics  -  12  days  (72  hours).  Environmental  -  15  days 
(90  hours),  and  Pneudraulics  -  12  days  (72  hours).  Each  course  focuses  exclusively  on  F-16 
subsystems  relevant  to  that  AFS. 

The  two  questions  we  asked  at  the  FTD  school  concerning  course  content  were  each  broken 
down  into  two  parts,  dealing  with  two  types  of  training.  The  first  part  focused  on  initial  and 
upgrade  training,  which  aim  to  increase  a  technician's  skill  level.  The  second  part  focused  on 
qualification  training,  where  the  goal  is  to  familiarize  a  technician  with  a  different  aircraft  or 
subsystem.  Since  the  course  content  for  these  two  types  of  training  was  found  to  be  very  similar, 
the  data  for  both  of  them  are  presented  together.  We  also  averaged  across  AFSs  before 
presenting  the  data  since  there  were  only  one  or  two  respondents  in  each  AFS.  For  the  first 
question  on  course  content,  which  focused  on  the  reference  tasks  in  Table  1,  the  most  frequent 
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responses  were  given  for  system/subsystem  knowledge  (91%)  and  equipment/tool  familiarization 
(82%).  Other  results  are  presented  in  Table  8,  which  shows  that  there  is  more  emphasis  on 
equipment/tool  familiarization  in  FTD  than  in  Technical  School. 

The  second  question  on  course  content  asked  instmctors  what  kind  of  troubleshooting 
knowledge  they  taught  (with  1  indicating  no  coverage,  3  moderate  coverage,  and  5  extensive 
coverage).  _As  Table  9  indicates,  the  results  for  this  question  are  similar  to  those  for  the 
Technical  School.  FTD  instructors  are  aware  of  and  claim  to  teach  all  of  the  kinds  of  knowledge 
needed  for  expert  troubleshooting.  However,  they  place  the  greatest  emphasis  on  mental-model 
knowledge  (coverage  of  4.7  for  device  knowledge  and  5.0  for  topographic/schematic 
knowledge). 

As  in  the  Technical  School,  a  variety  of  teaching  methods  are  used  in  the  FTD  school,  as 
shown  in  Table  10.  Again,  the  two  most  frequently  used  methods  (averaging  across  training  type 
and  AFS)  were  formal  training/lecture  (100%),  and  demonstration  and  practice  (95%).  Thus, 
FTD  instructors  mix  lecturing  with  some  of  the  aspects  of  apprenticeship  training,  particularly 
modeling  (demonstration)  and  practice. 

Another  similarity  to  Technical  School  is  that  simulators  and  aircraft  are  used  in  the  FTD 
classes  at  Hill  AFB,  although  the  Avionics  classes  have  no  simulator.  The  simulators  in  FTD 
classes  are  used  in  a  similar  manner  to  those  in  Technical  School.  Students  work  in  pairs.  The 
simulator  exercises  focus  on  use  of  the  TOs.  One  FTD  instructor  mentioned  that  using  the 
simulators  "teaches  the  students  to  trust  the  TOs."  However,  instmctors  sometimes  ask  students 
questions  that  require  them  to  go  beyond  the  procedures  knowledge  in  TOs  and  access  mental- 
model  knowledge.  For  example,  they  may  ask  students  to  explain  why  a  test  yielded  a  certain 
result.  Another  point  of  similarity  to  Technical  School  is  that  students  in  the  FTD  spend  very 
limited  time  on  the  simulators.  At  times,  operational  aircraft  are  used  for  training.  However,  this 
is  sporadic  and  depends  on  the  availability  of  aircraft  and  the  reliability  of  the  various  systems, 
i.e.,  whether  there  is  a  problem  to  troubleshoot. 

An  interesting  difference  was  found  in  the  way  simulators  are  used  in  the  classes  for  different 
AFSs  at  the  Hill  AFB  FTD.  As  mentioned  before,  the  FTD  Avionics  classes  have  no  simulator. 
The  simulators  for  the  Electrical  and  Environmental  classes  are  used  regularly.  However,  the 
simulator  for  the  Pneudraulics  classes,  though  available,  is  not  used  because  it  is  out  of  date. 
Students  in  these  classes  are  able  to  practice  troubleshooting  only  in  the  rare  case  of  a  fault 
occurring  when  they  are  practicing  an  operational  checkout  on  the  training  aircraft.  This 
situation  highlights  the  importance  of  simulators  to  classroom  troubleshooting  instruction,  and 
the  difficulty  of  updating  expensive,  high-fidelity  simulators  like  the  flatboards. 

Table  11  shows  that  FTD  students  are  most  frequently  evaluated  via  actual  equipment 
performance  (100%),  and  through  the  use  of  checklists  (86%).  These  data  suggest  that  in 
evaluating  students,  FTD  instructors  rely  less  on  written  tests  and  more  on  actual  equipment 
performance,  as  compared  to  Technical  School  instructors.  Typically,  trainees  are  certified  and 
decertified  by  the  same  method  and  forms  for  both  initial/upgrade  and  qualification  training.  The 
method  usually  involves  the  trainee  demonstrating  proficiency  through  task  performance.  The 
instructor  then  manually  records  this  status  on  various  forms.  AF  Forms  325  and  623  are  used  as 
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well  as  documentation  through  the  Core  Automated  Maintenance  System  (CAMS).  These 
records  are  then  carried  through  to  the  work  center  and  maintained  by  the  supervisor. 

Logistics  Support  Training 

Currently,  at  Hill  ATO,  the  LST  is  conducting  training  similar  to  the  FTD.  Typically  the  LST 
supplements  FTD  training  or  provides  the  training  that  the  FTD  is  not  capable  of  providing.  One 
of  the  main  functions  of  the  LST  is  to  coordinate  training  between  the  workcenters  and  the  FTD. 
This  training  has  usually  been  requested  or  is  required  for  the  various  workcenters;  LST 
coordinates  the  scheduling  of  personnel  for  FTD  courses.  One  LST  instructor  was  interviewed, 
using  the  same  questionnaire  as  for  the  FTD  instructors.  For  this  reason,  and  those  mentioned 
above,  data  for  this  individual  was  combined  with  that  of  the  FTD  instructors. 

On-the-Job-Training 

We  collected  data  on  OJT  for  the  Avionics,  Electrical  and  Engineering,  and  Pneudraulics 
AFSs  at  Hill  AFB.  Before  trainees  begin  OJT,  their  supervisors  assess  their  knowledge  and  skills 
through  verbal  questioning,  by  observing  their  performance,  and  by  checking  their  training 
history  in  their  OJT  folder.  Based  on  this  information,  supervisors  then  determine  what  tasks  or 
skills  the  trainee  needs  to  learn  and  records  this  on  the  basic  OJT  record,  AF  Form  623.  The 
tasks  to  be  trained  are  taken  primarily  from  the  STS,  which  generally  lists  the  skills  and  tasks 
needed  for  performance  in  a  particular  AFS.  For  example,  the  STS  for  AFSC  452X2,  Avionics 
contains  tasks  such  as  "isolate  malfunctions  to  fuel  flow  indicating  system"  and  "use  torque 
indicating  devices".  Tasks  to  be  trained  also  include  other  maintenance-related  tasks  such  as 
those  peculiar  to  the  local  base  and  maintenance  facility.  OJT  is  administered  on  the  flightline  or 
off-flightline  in  hangers  by  supervisors  or  an  assigned  trainer. 

The  questionnaire  data  presented  below  is  based  on  interviews  conducted  with  supervisors. 
The  data  showed  that  the  duration  of  OJT  varies  widely  depending  on  the  task,  ranging  from  a 
few  hours  to  months.  A  particular  task  is  usually  trained  between  one  to  three  times  per  week; 
the  duration  of  OJT  sessions  is  between  1  and  8  hours. 

As  at  the  FTD,  the  questions  concerning  the  content  of  OJT  were  asked  for  both 
initial/upgrade  and  qualification  training  and  for  the  three  AFSs  noted  above.  Only  the  overall 
data  is  presented  below.  For  the  question  on  OJT  content  that  focused  on  the  reference  tasks  in 
Table  1,  the  most  frequent  responses  were  given  to  subtasks  (81%),  system/subsystem  knowledge 
(81%),  and  equipment/tool  familiarization  (78%).  The  remainder  of  the  results  are  shown  in 
Table  8.  Naturally  enough,  there  is  less  emphasis  on  theoretical  knowledge  (system/subsystem 
knowledge)  in  OJT,  and  more  emphasis  on  job-related  tasks  (subtasks)  and  procedures. 

The  second  question  on  OJT  content  focused  on  what  kind  of  troubleshooting  knowledge  is 
taught.  The  results  ,  shown  in  Table  9,  indicate  that  supervisors  are  aware  of  and  give  at  least 
moderate  coverage  to  all  of  the  kinds  of  knowledge  needed  for  expert  troubleshooting. 

The  teaching  methods  used  in  OJT  are  different  than  those  used  in  the  classroom 
environments.  The  most  frequently  used  methods  are  demonstration  and  practice  (79%),  and 
oversight  and  assistance  (78%).  Other  responses  are  included  in  Table  10.  Thus,  as  would  be 
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expected,  OJT  emphasizes  apprenticeship  training  (coaching/oversight,  independent  work  with 
checks)  more  than  Technical  School  or  FTD  training,  while  focusing  less  on  formal  methods  like 
lecturing.  In  addition.  Career  Development  Courses  are  used  during  OJT,  although  infrequently. 
A  number  of  OJT  supervisors  stressed  that  the  only  way  for  trainees  to  learn  many  maintenance 
tasks  was  through  hands-on  experience.  As  an  example  of  this,  one  respondent  mentioned  that  a 
certain  removal  and  replacement  operation  took  about  six  hours  the  first  time  it  was  done,  but 
could  be  doiie  in  45  minutes  with  practice.  In  line  with  this  emphasis  on  hands-on  training,  most 
OJT  takes  place  on  the  flightline. 

Evaluation  of  trainees  performance  during  OJT  takes  place  mostly  on  the  flightline,  as  well. 
The  results  shown  in  Table  11  indicate  that  trainees  are  most  frequently  evaluated  via  actual 
equipment  performance  (84%),  and  through  use  of  checklists  (43%).  Records  of  trainee  progress 
are  kept  on  Form  623.  Typically,  trainees  are  certified  and  decertified  by  the  same  methods  and 
forms  for  both  initial/upgrade  and  qualification  training.  The  method  usually  involves  the  trainee 
demonstrating  proficiency  of  STS  objectives  through  task  performance.  The  supervisor  or  trainer 
manually  records  this  status  on  AF  Forms  623  and  797,  as  well  as  in  CAMS.  These  records  are 
then  maintained  by  the  supervisor. 

3.5.3  Summary  of  the  Air  Force  Maintenance  Training  Process 

We  collected  data  in  three  training  environments  on  the  content  of  maintenance  training  (both 
general  and  specific  to  troubleshooting),  and  on  training  and  evaluation  methods.  Concerning  the 
general  content  of  maintenance  training  (see  Table  8),  the  Technical  School  teaches  more 
systera/subsystem  knowledge  than  OJT,  whereas  OJT  focuses  on  more  detailed,  installation- 
specific  knowledge  concerning  specific  subtasks  and  extenuating  flightline  procedures.  FTD 
training  falls  between  these  two  extremes.  Instructors  in  all  three  training  environments  claim  to 
teach  all  of  the  aspects  of  expert  troubleshooting  knowledge  outlined  in  Table  2  (see  Table  9), 
with  a  slight  tendency  towards  emphasizing  mental-model  knowledge  (i.e.,  specific  device 
knowledge  and  schematics).  As  expected,  the  classroom  environments,  FTD  and  Technical 
School,  emphasize  formal  training  and  evaluation  methods  such  as  lecturing  and  written  tests 
(see  Tables  10  and  11).  OJT  insfructors,  in  contrast,  focus  more  on  apprenticeship-training 
methods  such  as  independent  work  with  oversight,  and  evaluate  trainees  more  often  via  actual 
equipment  performance. 

It  appears  evident  that  timely  evaluation  and  feedback  during  and  after  training  is  mostly 
dependent  on  the  availability  of  the  training  supervisor  or  instructor.  In  the  classroom  settings,  to 
receive  feedback  or  evaluative  information  during  training,  the  student  must  wait  for  the 
instmctor's  availability  to  answer  questions  or  provide  written  test  critiques.  As  for  OJT,  it 
appears  that  feedback  and  evaluative  information  is  typically  forthcoming  with  little  delay. 
Remedial/review  training  in  all  environments  typically  consists  of  the  supervisor  or  instructor 
going  back  over  the  material,  task  or  skill  that  should  have  been  mastered  providing  more  in- 
depth  information.  Students  are  then  re-evaluated  for  a  second,  and  final,  time. 

Overall,  this  is  not  a  surprising  picture.  The  classroom  environments  use  more  formal 
training  and  evaluation  methods,  while  OJT  emphasizes  hands-on  training.  We  would  like  to 
stress  that  OJT  instructors,  and  to  some  extent  other  instmctors,  believe  very  strongly  in  the  value 
of  hands-on  training  as  the  primary  way  to  leam  maintenance  skills.  However,  the  Technical 
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School  and  LST  are  the  only  training  environments  where  aircraft  are  dedicated  for  purely 
training  purposes,  and  even  there  hands-on  training  is  extremely  limited.  For  FTD  and  OJT, 
hands-on  training  is  largely  dependent  on  the  availability  of  aircraft  for  training  and  on  the 
reliability  of  the  aircraft  under  study. 

Where  hands-on  access  to  aircraft  is  limited,  as  in  the  classroom  environments,  instructors 
rely  on  simulators  instead.  The  classroom  instructors  we  interviewed  saw  the  simulators  as  a 
very  important  part  of  instruction.  Some  instructors  both  at  Technical  School  and  FTD  reported 
using  the  simulators  for  30  to  35%  of  class  time.  Although  individual  students  have  limited 
access  to  the  simulators  because  only  two  students  can  use  a  simulator  at  a  time,  this  is  still  a 
significant  portion  of  class  time  to  devote  to  a  single  instructional  method. 

Technical  School  and  FTD  instructors  use  an  apprenticeship-training  approach  with  the 
simulators,  allowing  the  students  to  work  on  the  problem  independently  most  of  the  time,  but 
providing  coaching  when  necessary.  Instructors  reported  using  the  simulators  in  ways  that  would 
teach  some  of  the  problem-solving  aspects  of  troubleshooting.  For  example,  they  asked  students 
to  explain  why  a  troubleshooting  test  gave  a  certain  result,  and  they  had  students  locate  the  tests 
and  replacements  they  made  on  a  system  schematic. 

On  the  other  hand,  we  noticed  three  significant  problems  that  limit  the  usefulness  of  the 
current  simulators  in  teaching  the  problem-solving  aspects  of  troubleshooting.  First,  individual 
student  access  to  the  simulators  is  limited.  Second,  the  flatboard  simulators  are  hard  to  update 
and  may  not  be  used  at  all  when  they  become  outdated.  Third,  the  current  simulators  only  give 
students  problems  requiring  routine  use  of  the  TOs. 

3.5.4  Conclusions  Concerning  the  Maintenance  Training  Process 

Given  our  findings  concerning  maintenance  training  in  the  classroom  and  OJT  environments, 
how  might  IMIS  fit  into  these  training  environments?  We  will  give  a  brief  answer  to  this 
question  here,  and  elaborate  on  these  ideas  in  the  concluding  sections  of  this  report.  With  respect 
to  the  classroom  environments,  using  IMIS  as  a  simulator  holds  considerable  promise. 
Instructors  value  the  current  simulators  and  use  them  to  teach  problem-solving  aspects  of 
troubleshooting.  IMIS  simulators  could  be  used  in  the  classroom  in  addition  to  the  flatboard 
simulators.  This  could  alleviate  the  problems  with  the  current  simulators  by  giving  students 
more  access  to  simulators  and  allowing  easy  updating  of  simulators.  In  addition,  IMIS 
simulators  could  give  students  problems  requiring  them  to  go  beyond  the  TOs,  while  still  using 
the  intelligence  of  the  IMIS  DM  to  generate  advice  and  feedback. 

With  respect  to  OJT,  supervisors'  and  instructors'  emphasis  on  hands-on  training  may  limit 
the  use  of  IMIS  as  a  stand-alone  simulator  (i.e.,  apart  from  the  aircraft).  However,  IMIS  could 
be  used  as  a  training  aid  during  actual  maintenance  activities.  For  example,  if  a  trainee  was 
repairing  an  aircraft  without  being  pressed  for  time,  IMIS  could  be  put  into  a  training  mode 
where  it  would  do  such  things  as  give  more  elaborate  explanations  for  its  choices  and  ask  the 
trainee  to  predict  the  results  of  tests  (Babiarz  et  al.,  1989). 
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IV.  IMPLICATIONS  AND  HYPOTHESES 
CONCERNING  THE  USE  OF  IMIS  IN  TRAINING 


We  now  consider  the  implications  of  the  Training-Situation- Analysis  findings  for  how  IMIS 
could  be  used  in  training,  and  suggest  hypotheses  for  how  IMIS  may  affect  maintenance  training. 
This  section  is  structured  in  terms  of  two  questions:  1)  What  kind  of  maintenance  skills  need  to 
be  trained?  and  2)  How  can  IMIS  help  train  these  skills?  Specific  hypotheses  and  implications 
are  italicized  in  the  text  and  listed  in  Table  12  at  the  end  of  this  section. 

4.1  Maintenance  Training  Needs 

Concerning  the  type  of  maintenance  skills  to  be  trained,  the  findings  on  the  current 
maintenance  situation  suggest  that  technicians  need  training  in:  1)  the  procedures  knowledge 
contained  in  the  TOs,  and  2)  the  problem-solving  knowledge  needed  to  handle  more  difficult 
maintenance  problems  that  require  more  than  TO  knowledge.  Most  maintenance  activities 
involve  following  pre-leamed  procedures.  However,  a  substantial  percentage  of  maintenance 
tasks  (about  30%  of  the  troubleshooting  tasks  at  Hill  AFB)  require  problem-solving  knowledge 
beyond  that  contained  in  the  TOs.  Thus,  technicians  need  training  in  a  wide  variety  of 
procedures  (e.g.,for  equipment  use,  safety,  recordkeeping,  and  troubleshooting).  Also,  in  order 
to  solve  difficult  troubleshooting  problems,  technicians  need  explicit  training  in  all  of  the 
different  kinds  of  problem-solving  knowledge  used  by  expert  troubleshooters,  specifically: 
mental-model  and  symptom-fault-association  knowledge,  procedures,  strategies,  and 
coordination  processes. 

4.2  How  IMIS  Can  Assist  in  Maintenance  Training 

How,  then,  can  IMIS  help  to  train  this  diverse  array  of  skills?  In  answering  this  question,  we 
will  consider  each  phase  of  the  instructional  process,  including  instructional  design,  instructional 
delivery,  performance  assessment,  course  management,  and  recordkeeping.  We  will  present 
implications  and  hypotheses  for  each  of  these  phases.  For  ease  of  presentation,  we  will  focus  on 
the  instruction  and  assessment  phases  first. 

4.2.1  Instructional  Uses 

We  feel  that  the  most  effective  instructional  use  of  IMIS  is  as  a  simulator,  and  so  we  will 
concentrate  on  this  use.  There  are  two  reasons  for  this  focus.  First,  as  a  job  aid,  IMIS  provides 
an  interface  (the  PMA)  that  coordinates  the  entire  maintenance  process  for  the  technician. 
Therefore,  it  would  be  very  easy  to  implement  realistic  simulations  using  the  PMA.  Second,  we 
feel  that  an  IMIS  simulator  could  be  of  significant  help  in  training  the  most  difficult  skills 
technicians  have  to  learn  —  the  problem-solving  skills  required  for  expert  troubleshooting. 

The  apprenticeship-training  literature  suggests  that  coached  practice  is  essential  for  learning 
complex  problem-solving  skills.  OJT  supervisors  we  interviewed  also  stressed  the  importance  of 
practice  in  training  maintenance  skills.  However,  our  data  shows  that  in  Technical  School  and 
FTD,  students  have  very  little  chance  to  practice  their  skills  on  aircraft  or  simulators.  As  we  will 
describe  below,  an  IMIS  simulator  can  provide  extensive  coached  practice  on  troubleshooting 
problems,  thus  meeting  a  crucial  training  need. 
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Technicians  also  need  training  in  many  procedures  that  do  not  involve  complex  problem¬ 
solving.  Furthermore,  some  of  these  procedures  (e.g,,  removing  and  replacing  components,  using 
a  multimeter,  etc.)  involve  perceptual  judgments  that  are  hard  to  teach  with  a  simulator.  This  is 
probably  the  reason  why  OJT  supervisors  favor  hands-on  aircraft  training.  We  feel  that  IMIS 
would  be  less  effective  at  training  perceptual-based  procedures,  at  least  when  it  is  used  in  a 
stand-alone  mode.  Some  of  the  problems  of  teaching  the  perceptual  aspects  of  maintenance  can 
be  solved  by  using  an  IMIS  simulator  with  the  aircraft  or  other,  more  realistic  simulators. 
Therefore,  we  hypothesize  that  an  IMIS  simulator  can  help  teach  maintenance  skills  in  three 
dijferent  configurations:  with  the  aircraft,  with  the  flatboard  simulators,  and  in  a  stand-alone 
configuration. 

When  an  aircraft  is  available  for  training,  or,  during  OJT,  when  a  trainee  is  working  on  an 
aircraft  without  time  pressure,  IMIS  could  be  run  in  a  training  mode.  This  configuration  would 
closely  simulate  flightline  conditions,  while  providing  more  extensive  coaching  and  data 
recording  than  is  usually  done  by  IMIS.  When  MIS  is  used  with  the  flatboards,  the  simulated 
instrument  panels  and  video  displays  of  aircraft  components  would  help  train  the  perceptual 
aspects  of  maintenance  procedures.  When  MIS  is  used  in  a  stand-alone  simulation,  less  realism 
would  be  provided.  However,  we  feel  that  even  in  the  stand-alone  configuration,  MIS 
simulations  could  still  provide  effective  maintenance  training.  One  reason  supporting  this 
hypothesis  is  that  some  of  the  important  aspects  of  troubleshooting  skill,  as  outlined  in  Table  2, 
are  cognitive,  rather  than  perceptual,  in  nature.  All  of  the  types  of  troubleshooting  knowledge  in 
Table  2  include  cognitive  aspects  that  can  be  practiced  in  a  simulator,  away  from  the  aircraft. 
Another  reason  for  advocating  off-aircraft  simulators  is  the  effectiveness  of  other  computer- 
based  troubleshooting  simulators  such  as  Sherlock  (Lajoie  &  Lesgold,  1989). 

Another  advantage  for  using  MIS  simulators  in  the  three  configurations  described  above  is 
that  this  allows  MIS  to  be  used  in  all  Air  Force  training  environments.  Technical  School,  FTD, 
and  OJT.  Using  IMIS  in  multiple  training  contexts  follows  one  of  the  principles  of 
apprenticeship  training  and  can  increase  transfer  of  training. 

The  MIS  simulator  we  describe  below  focuses  on  teaching  the  problem-solving  skills  needed 
for  expert  troubleshooting.  We  focus  on  complex  troubleshooting  rather  than  other  maintenance 
procedures  for  two  reasons.  First,  teaching  troubleshooting/problem-solving  skills  effectively  is 
an  important  and  unfulfilled  need.  Second,  MIS  is  in  a  better  position  to  provide  effective 
training  in  troubleshooting  and  problem  solving  than  other  training  environments  (e.g.,  authoring 
systems  for  computer-based  training)  because  an  MIS  training  environment  has  access  to  the 
intelligence  and  knowledge  in  the  DM  and  CDM. 

We  envision  two  levels  of  detail  in  an  MIS  simulation.  The  first,  which  we  call  a  detailed 
simulation,  involves  using  the  APS  to  augment  the  CDM  with  instructional  information  for  each 
aircraft  sub-system.  This  would  use  the  full  capabilities  of  IMIS  to  present  maintenance  training. 
However,  authoring  and  updating  the  instructional  information  needed  for  a  detailed  simulation 
could  prove  expensive,  especially  if  IMIS  is  used,  as  planned,  for  multiple  aircraft.  This  led  us  to 
consider  a  second  level  of  detail  in  simulation,  what  we  call  generic  simulation.  This  would  use 
minimal  authoring  of  instructional  materials,  instead  relying  on  knowledge  and  data  already  in 
the  DM  and  the  CDM.  As  we  mentioned  previously,  in  its  job-aid  capacity,  MIS  contains 
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considerable  knowledge  about  aircraft  systems  and  maintenance  processes.  This  information 
overlaps  considerably  with  the  knowledge  used  by  expert  human  troubleshooters.  As  will  be 
described  below,  we  feel  that  the  information  currently  in  the  CDM  is  enough  to  create  a 
powerful  simulation  environment,  if  it  is  presented  to  students  using  appropriate  instructional 
strategies.  The  generic  simulation  would  require  minimal  authoring  and  updating  beyond  that 
needed  to  create  the  CDM.  Once,  initially  developed,  the  generic  simulation  capability  would  be 
available  at  Jittle  extra  cost  for  any  aircraft  with  a  CDM  that  can  support  IMIS  job  aiding.  Of 
course,  more  detailed  instructional  information  can  be  added  to  the  generic  simulation. 
However,  since  the  generic  simulation  has  the  potential  for  being  an  inexpensive  and  effective 
instructional  tool,  we  recommend  developing  it  first,  and  will  describe  it  first  in  this  section. 

In  the  following,  we  will  outline  the  features  that  could  be  included  in  a  generic  IMIS 
simulation.  To  increase  its  instructional  effectiveness,  this  simulator  should  follow  the  principles 
of  apprenticeship  training  (practice  on  realistic  problems,  coaching,  fading,  and  collaborative 
learning).  The  first  feature,  already  discussed  here,  involves  allowing  students  extensive 
problem-solving  practice.  However,  if  this  practice  is  to  be  beneficial,  the  kind  of  problems 
students  practice  on  is  important.  Initially,  students  need  to  practice  problems  that  involve 
routine  use  of  the  TOs,  as  is  done  with  the  flatboard  simulators.  This  will  teach  students 
procedures  knowledge.  Later,  students  need  to  practice  difficult  problems  that  cannot  be  solved 
solely  by  the  TOs.  This  will  require  them  to  learn  and  use  the  other  kinds  of  knowledge  needed 
for  expert  troubleshooting.  The  capabilities  of  the  IMIS  DM  potentially  can  allow  it  to  solve 
some  problems  that  the  TOs  alone  cannot  solve.  However,  as  mentioned  earlier,  IMIS's  ability  to 
provide  more  accurate  troubleshooting  advice  than  the  TOs  has  not  yet  been  convincingly 
demonstrated.  If  this  ability  is  demonstrated,  it  would  be  possible  for  an  TMTS  simulator  to 
generate  difficult  ("beyond  the  TOs")  problems  and  use  the  intelligence  in  the  DM  to  coach 
students  on  these  problems. 

A  second  principle  of  cognitive  apprenticeship  involves  coaching  students  as  they  solve 
problems.  Coaching  includes  modeling  expert  problem-solving  behavior  and  thinking,  as  well  as 
giving  feedback  in  the  form  of  questions,  hints,  and  reminders.  We  think  that  IMIS  can  provide 
this  kind  of  coaching,  using  the  knowledge  in  the  DM  and  the  CDM.  Furthermore,  this  coaching 
can  help  students  learn  most  of  the  knowledge  needed  for  expert  troubleshooting  (see  Table  2. 
For  example,  when  the  DM  uses  the  half-split  strategy  to  choose  a  test,  the  coach  could  describe 
this  strategy  to  smdents.  This  is  an  example  of  modeling.  Later,  the  simulation  could  give 
students  the  chance  to  choose  their  own  tests  and  replacements,  before  seeing  the  DM's 
recommendations.  At  this  point,  the  coach  could  determine  whether  the  tests  a  student  chose 
reflect  use  of  half  split.  If  they  do  not,  the  coach  could  remind  the  students  of  the  strategy. 

Another  example  of  coaching  involves  teaching  symptom-fault  associations.  When  choosing 
a  test,  the  coach  could  model  symptom-fault-association  knowledge  by  pointing  out  that  a  certain 
test  was  not  chosen  because  it  had  a  very  high  probability  of  passing  (say,  90%)3 .  Thus,  this  test 
would  not  yield  as  much  information  as  a  test  that  has  closer  to  a  50%  probability  of  passing. 
Later,  when  students  choose  their  own  tests,  the  coach  could  ask  them  to  enter  the  expected 
probabilities  that  these  tests  will  pass,  and  correct  them  if  they  are  wrong. 


^  This  type  of  modeling  was  implemented  in  a  prototype  IMIS  instructional  simulation  developed  by  Babiarz,  et  al. 
(1989). 
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These  two  examples  reflect  a  general  instructional  strategy  of  first  having  the  DM  model 
appropriate  troubleshooting  knowledge  as  the  student  is  solving  a  problem.  Later,  the  student  is 
given  more  responsibility  for  decisions  during  problem  solving  (e.g.,  choosing  their  own  tests). 
At  this  point,  through  questions  and  feedback,  the  coach  focuses  the  students  on  the  appropriate 
troubleshooting  knowledge  for  the  current  decision.  This  instructional  strategy  could  be  used  to 
instruct  most  of  the  knowledge  in  Table  2,  in  particular,  symptom-fault  associations,  procedures, 
strategies,  and  some  coordination  processes.  As  mentioned  earlier,  instructing  students  in 
mental-model  knowledge  in  a  generic  simulation  (i.e.,  without  adding  information  to  the  CDM) 
would  be  difficult,  since  the  CDM  contains  little  mental-model  knowledge. 

The  coaching  described  here  implements  some  aspects  of  an  ITS,  without  using  a  student 
model.  Instead  of  basing  its  response  on  a  model  of  student  abilities,  the  coach  determines  its 
response  by  comparing  the  student's  and  the  expert's  (i.e.,  the  DM's)  performance.  Newman 
(1989)  has  described  a  similar  ITS  that  does  not  involve  student  modeling.  In  the  assessment 
section,  below,  we  will  outline  how  some  student  modeling  capabilities  could  be  incorporated  in 
the  MIS  generic  simulation. 

The  effectiveness  of  the  coaching  described  above  depends,  to  some  extent,  on  whether 
Evas  can  solve  some  of  the  difficult  troubleshooting  problems  that  the  TOs  cannot  solve.  If 
IMIS  cannot  do  this,  its  coaching  will  be  less  effective,  although  not  completely  ineffective. 
Some  of  the  coaching  described  above  (e.g.,  modeling  troubleshooting  knowledge  and  skills) 
might  still  help  students  learn,  even  if  IMIS  is  not  able  to  solve  some  problems.  The  extent  to 
which  the  effectiveness  of  IMIS's  coaching  depends  on  the  accuracy  of  its  troubleshooting  advice 
will  be  investigated  further  in  the  next  phase  of  this  project. 

A  third  principle  of  cognitive  apprenticeship  is  fading,  whereby  the  coach  withdraws  support 
as  the  student  becomes  more  proficient.  A  simple  way  to  implement  fading  would  be  to  have 
different  levels  of  coaching  in  the  generic  simulation.  The  novice  level  would  focus  on  the  coach 
modeling  and  explaining  the  troubleshooting  process,  with  the  students  having  little  input  into 
the  decision  making.  An  intermediate  level  would  allow  students  to  have  more  input  into 
decisions  during  problem  solving,  but  the  coach  would  ask  questions  and  give  feedback  to  ensure 
they  were  using  the  correct  knowledge  in  making  those  decisions.  The  expert  level  would 
provide  no  coaching,  allowing  the  students  to  make  troubleshooting  decisions  as  if  they  were 
using  IMIS  on  the  flightline.  The  instructor  or  the  students  could  choose  which  level  of  coaching 
to  use  for  a  particular  problem. 

An  even  more  effective  way  to  implement  fading  would  involve  some  student  modeling.  As 
will  be  described  in  the  assessment  section,  a  diagnostic  capability  could  be  added  to  the  generic 
simulation,  giving  it  the  ability,  for  example,  to  evaluate  and  remember  how  well  a  student 
knows  the  half-split  strategy.  Given  this  information  in  a  student  model,  the  coach  could 
determine  what  kind  of  coaching  to  give  concerning  this  strategy.  A  student  modeling  capability 
would  allow  more  tailoring  of  instruction  to  individual  needs  than  simply  having  novice, 
intermediate,  and  expert  simulations,  but  would  be  more  costly  to  develop. 

A  fourth  principle  of  cognitive  apprenticeship  is  to  encourage  collaborative  problem-solving 
among  students.  This  principle  is  already  being  followed  in  the  use  of  the  flatboard  simulators  in 


34 


Technical  School  and  the  FTD.  Students  solve  problems  in  pairs  with  the  instructor  occasionally 
offering  assistance  or  coaching.  We  recommend  that  this  practice  continue  with  IMIS  simulators. 
In  addition,  other  forms  of  collaboration  are  possible  (Katz  &  Lesgold,  in  press).  For  example, 
students  could  pose  problems  for  each  other.  Or,  a  student's  performance  on  a  troubleshooting 
problem  could  be  recorded  using  IMIS's  recording  capabihty  and  later  replayed  for  discussion  by 
other  students  or  the  instructor.  With  appropriate  guidance  by  an  instructor,  a  classroom  or 
learning  laboratory  with  a  number  of  IMIS  simulators  could  become  a  rich  environment  for 
student  interaction  and  learning. 

All  of  the  instructional  features  described  above  could  be  implemented  in  a  generic  IMIS 
simulator  that  uses  just  the  CDM  knowledge  needed  for  job  aiding.  The  detailed  simulator  could 
build  on  these  features  in  a  number  of  ways.  For  example,  the  APS  could  be  used  to  add  mental- 
model  information  to  the  CDM  so  that  students  could  be  coached  in  this  kind  of  knowledge. 
Also,  videodisk  or  virtual-reality  capabilities  could  be  added  to  the  simulator.  This  would  be  of 
particular  help  in  teaching  students  some  of  the  perceptual  aspects  of  maintenance  activities.  For 
example,  when  removing  and  replacing  a  component  as  part  of  a  troubleshooting  problem, 
students  could  use  the  videodisk  screen  to  see  what  the  component  looks  like  and  where  it  is 
located  on  the  aircraft.  A  videodisk  capability  like  this  is  already  implemented  for  the  flatboard 
simulators  used  in  Technical  School  and  the  FTD. 

4.2.2  Assessment  Uses 

An  IMIS  simulator  could  perform  two  types  of  student  assessment:  I )  detailed  diagnosis  of 
students'  strengths  and  weaknesses  for  use  by  an  instructor,  and  2)  more  general  evaluation  of 
student  readiness  for  course  advancement  or  particular  work  tasks.  We  will  describe  the 
diagnostic  assessment  first. 

As  we  mentioned  earlier,  a  generic  IMIS  simulator  could  do  some  simple  student  modeling. 
For  example,  the  simulator  could  obtain  information  about  a  student's  use  of  the  half-split 
strategy  by  having  the  smdent  choose  troubleshooting  tests  and  replacements  before  seeing  the 
DM's  recommendations  (e.g.,  as  in  the  intermediate  level  simulator  described  above).  Using 
information  available  in  the  CDM,  the  simulator  could  immediately  evaluate  each  proposed  test 
and  replacement  in  terms  of  how  well  it  conformed  to  the  half-split  strategy.  This  information 
could  be  used  to  update  a  variable  in  a  student  model  representing  knowledge  of  this  strategy.  A 
similar  approach  could  be  used  to  model  student  knowledge  of  other  strategies  (e.g.,  elimination), 
symptom-fault  associations,  and  some  coordination  processes.  This  modeling  could  be  done  as 
part  of  the  generic  IMIS  simulator,  since  it  relies  on  knowledge  currently  in  the  CDM.  Only 
limited  mental-model  knowledge  could  be  modeled  using  information  currently  in  the  CDM. 

The  information  in  a  student  model  could  be  used  in  two  ways.  First,  the  simulator  could  use 
this  information  to  tailor  its  coaching  to  individual  student  needs,  as  students  work  on  problems. 
This  would  involve  implementing  the  capabilities  of  a  full-fledged  ITS.  We  think  this  is 
possible,  given  the  current  configuration  of  the  DM  and  the  CDM,  although  the  student  model 
would  be  limited,  in  that  it  would  not  represent  all  the  knowledge  required  for  expert 
performance.  A  second,  and  less  costly,  way  to  use  the  detailed  diagnostic  information  in  a 
student  model  would  be  to  print  out  a  report  describing  a  student's  knowledge.  The  instructor 
could  use  this  report  to  plan  instruction,  and/or  students  could  use  it  to  plan  their  studying. 
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A  second  kind  of  assessment  that  could  be  provided  by  an  IMIS  simulator  is  a  general 
evaluation  of  student  readiness  for  course  advancement  or  job  performance.  For  example,  a 
supervisor  could  assign  a  student  to  practice  a  certain  troubleshooting  task  on  the  simulator. 
When  the  student  is  ready,  he  or  she  could  take  a  test  run  through  the  simulator  on  that  task.  The 
simulator  could  then  print  out  a  report  comparing  the  student  and  the  simulated  expert  (i.e.,  the 
DM)  on  information  such  as  the  number  of  tests  and  component  replacements  performed  and 
simulated  maintenance  time.  This  information  could  help  the  supervisor  decide  whether  the 
student  was  ready  to  perform  this  task  on  the  flightline.  Such  a  tool  in  the  Technical  School 
could  enable  students  to  achieve  higher  proficiency  levels  prior  to  graduation  or  to  reduce  the 
time  required  to  achieve  current  proficiency  levels. 

The  above  assessment  functions  could  be  performed  by  a  generic  IMIS  simulator,  with 
minimal  changes  to  the  CDM.  A  more  detailed  simulation  could  focus  on  functions  such  as 
assessing  students'  mental-model  knowledge. 

To  summarize  our  ideas  concerning  an  IMIS  simulator,  a  generic  IMIS  simulator  could  be 
developed  with  the  instructional  and  assessment  capabilities  described  above  (i.e.,  extended 
problem-solving  practice,  coaching,  fading,  collaboration,  assessment  reports,  and  limited 
student  modeling).  This  simulator  could  use  the  intelligence  and  information  currently  available 
in  the  DM  and  CDM,  and  thus  would  not  require  extensive  and  costly  authoring  of  instructional 
materials.  The  simulator  would  follow  effective  principles  of  instruction  (cognitive 
apprenticeship)  and  implement  some  features  of  an  ITS.  We  hypothesize  that  such  a  simulator 
would  have  a  number  of  beneficial  effects,  including: 

•  increasing  student  practice  of  maintenance  tasks; 

•  increasing  student  knowledge  of  the  problem-solving  skills  necessary  for  expert 
troubleshooting  (cf.  Table  2); 

•  reducing  training  time  without  requiring  additional  instructors;  and/or  increasing 
students'  proficiency  levels. 

However,  some  of  the  troubleshooting  processes  used  by  the  current  version  of  IMIS, 
particularly  those  involving  mental-model  knowledge  and  coordination  processes,  are 
significantly  different  from  those  used  by  humans.  Therefore,  in  its  current  state,  IMIS  probably 
could  not  be  used  to  train  advanced  troubleshooting  skills  (e.g.,  for  skill-level  7  technicians). 
IMIS  currently  offers  more  potential  for  training  novice  technicians,  although  even  for  training 
novices,  the  addition  of  some  mental-model  knowledge  to  IMIS  is  recommended. 

Using  the  generic  IMIS  simulator  as  a  base,  a  more  detailed  simulator  could  be  developed 
with  increased  use  of  mental-model  knowledge  and  video-based  output  (video  or  virtual  reality). 

4.2.3  Course  Management  and  Recordkeeping 

The  scenario  just  discussed  —  of  students  using  a  simulator  to  practice  and  take  tests  on 
specific  maintenance  tasks,  with  instructors  receiving  evaluation  reports  -  brings  up  issues  of 
course  management,  task  proficiency,  and  recordkeeping.  Concerning  course  management,  IMIS 
could  be  the  central  tool  in  a  Personalized  System  of  Instruction  course  (Maddux,  Johnson  & 
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Willis,  1992).  In  this  system,  each  student  could  be  assigned  an  individualized  set  of 
maintenance  tasks  to  master.  They  could  practice  these  tasks  in  a  learning  laboratory,  which  has 
IMIS  simulators,  TOs,  and  other  reference  materials,  as  well  as  an  instructor  to  serve  as  a 
resource  person.  Whenever  students  used  the  IMIS  simulator  to  take  tests  on  particular 
maintenance  tasks,  IMIS  could  record  an  evaluation  of  their  performance.  It  could  also  keep 
track  of  students  progress  through  their  assigned  tasks.  If  this  laboratory  were  in  an  OJT  setting, 
the  data  concerning  trainee  task  proficiency  could  be  recorded  on  an  on-line  version  of  the  OJT 
progress  form  (AF  Form  623),  or  directly  in  BTS  .  This  type  of  laboratory  would  allow  self- 
paced  learning  and  could  be  used  for  initial  or  remedial  learning  of  maintenance  tasks. 


Through  its  function  of  recording  and  managing  student  training  data,  IMIS  can  link  up  with 
other  automated  training  systems,  such  as  the  Advanced  Training  System  (ATS)  for  use  in  the 
Technical  Schools  or  the  Base  Training  System  (BTS)  for  use  on  the  job.  By  tracking  student 
performance  on  various  tasks  while  in  Technical  School,  IMIS  could  feed  highly  accurate 
performance  data  to  ATS  for  use  in  evaluation  of  students.  IMIS  data  could  be  used  to  generate 
reports  of  overall  student  proficiency  by  STS  task,  or  to  produce  specific  training  deficiency 
reports  due  to  the  lack  of  some  training  resource.  The  same  kind  of  highly  accurate,  task  specific 
qualification  information  could  be  automatically  fed  into  BTS  by  IMIS.  This  would  alleviate  the 
need  for  the  OJT  supervisor  to  record  this  information  manually  or  enter  it  into  a  computerized 
system.  All  training  status  information  would  be  centrally  stored  on-line  where  higher  echelon 
personnel  would  have  access  to  it.  This  includes  information  such  as  which  maintenance 
personnel  are  fiilly  qualified  and  ready  for  deployment,  which  personnel  have  minor  deficiencies 
that  can  be  quickly  remedied,  and  what  tasks  need  remediation. 


4.2.4  Instructional  Design 


Instructional  design  and  authoring  would  be  necessary  to  develop  the  instructional  materials 
(e.g.,  materials  to  teach  mental-model  knowledge)  for  the  detailed  IMIS  simulator,  as  discussed 
above.  The  current  IMIS  system  for  authoring  is  the  APS.  It  is  possible  that  the  capabilities  of 
the  authoring/presentation  software  (APS)  may  be  augmented  by  linking  it  with  the  BTS,  ATS, 
or  Automated  Instructional  Design  Advisor  (AIDA)  to  serve  the  purpose  of  designing, 
developing,  delivering,  and  evaluating  maintenance  training.  Further  research  will  be  needed  to 
assess  efforts  at  integrating  or  interfacing  any  of  these  systems.  However,  we  currently  visualize 
these  interfaces  occurring  as  shown  in  Figure  3. 
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Figure  3.  Authoring  scenario  based  on  proposed  links  between  systems. 

The  authoring  scenario  depicted  in  Figure  3.  shows  that  performance  data  from  IMIS  could  be 
transferred  though  BTS  to  ATS.  Information  from  these  systems  would  then  be  used  in 
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identifying  tasks  which  need  training  support.  Such  data,  in  conjunction  with  expert  advice  from 
AIDA  could  then  be  used  to  author  instruction  for  presentation  on  IMIS  either  through  the  PMA 
or  some  stand  alone  computer  with  IMIS  data  available. 
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Table  12 
Final  Hypotheses 


Maintenance  Training  Needs 

Technicians  need  training  in:  1)  the  procedures  knowledge  contained  in  the  TOs,  and  2)  the  problem-solving 
knowledge  needed  to  handle  the  more  difficult  maintenance  problems  that  require  more  than  TO  knowledge. 

Technicians  need  training  in  a  wide  variety  of  procedures  (e.g.,  for  equipment  use,  safety,  recordkeeping,  and 
troubleshooting). 

Technicians  need  explicit  training  in  all  of  the  different  kinds  of  knowledge  used  by  expert  troubleshooters, 
spcifically:  mental-model  and  symptom-fault-association  knowledge,  procedures,  strategies,  and  coordination 
processes. 

How  IMIS  Can  Assist  in  Maintenance  Training 

Instructional  Uses 

The  most  effective  instructional  use  of  EMIS  is  as  a  simulator 

An  IMIS  simulator  can  effectively  teach  maintenance  skills  in  three  different  configurations:  with  the  aircraft,  with 
the  flatboard  simulators,  and  stand  alone.  This  would  allow  IMIS  to  be  used  in  multiple  training  contexts 
(Technical  School,  FID,  and  OJT),  which  can  increase  transfer  of  training. 

A  generic  IMIS  simulator  could  be  developed  with  both  instructional  and  assessment  capabilities,  including 
extended  problem-solving  practice,  coaching,  fading,  collaboration,  assessment  reports,  and  limited  student 
modeling.  This  simulator  could  use  the  intelligence  and  information  currently  available  in  the  DM  and  CDM,  and 
thus  would  not  require  extensive  and  costly  authoring  of  instructional  materials. 

The  generic  simulator  should  follow  principles  of  apprenticeship  training  (extensive  practice  on  realistic 
problems,  coaching,  fading,  collaborative  learning)  and  can  include  some  features  of  an  ITS. 

Practice:  Initially,  students  need  to  use  a  simulator  to  practice  problems  that  involve  routine  use  of  the  TOs, 
Later,  students  need  to  practice  difficult  problems  that  cannot  be  solved  solely  by  the  TOs. 

Coaching:  An  IMIS  simulator  can  provide  effective  coaching  (modeling,  questioning,  hinting,  reminding)  using 
just  the  knowledge  in  the  current  DM  and  CDM.  However,  the  effectiveness  of  IMIS's  coaching  may  be 
diminished  if  it  cannot  give  accurate  advice  on  difficult  troubleshooting  problems  that  the  TOs  cannot  solve. 

Fading:  Fading,  withdrawing  support  as  the  student  progresses,  can  be  implemented  by  having  different  levels  of 
coaching  (e.g.,  novice,  intermediate,  expert)  or  by  doing  limited  student  modeling. 

Collaborative  learning:  The  collaborative  learning  with  simulators  already  practiced  in  the  Technical  School  and 
FTD  should  be  continued  and  extended  with  IMIS  simulators. 

Assessment  Uses 

A  generic  IMIS  simulator  could  perform  two  types  of  student  assessment:  1)  detailed  diagnosis  of  students' 
strengths  and  weaknesses  for  use  by  an  instructor,  and  2)  more  general  evaluation  of  student  readiness  for  course 
advancement  or  particular  work  tasks. 

A  generic  IMIS  simulator  could  do  some  simple  student  modeling.  It  could  also  create  reports  containing 
information  concerning  student  knowledge  (e.g.,  knowledge  of  troubleshooting  strategies)  and/or  data  on  how 
well  students  performed  specific  maintenance  tasks. 
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Table  12.  continued 


Summary  of  Instructional  and  Assessment  Uses 

A  generic  simulator  using  the  instructional  and  assessment  features  described  above  would  have  some  or  all  of  the 
following  beneficial  effects: 

•  increasing  student  practice  of  maintenance  tasks; 

•  reducing  training  time  without  requiring  ihore  instructors; 

•  increasing  student  proficiency  levels;  and 

•  increasing  student  knowledge  of  the  problem-solving  skills  necessary  for  expert 
troubleshooting  (cf.,  Table  2). 

Through  its  function  of  recording  and  managing  student  training  data,  IMIS  can  link  up  with  other  automated 
training  systems,  such  as  the  Base  Training  System  (BTS).  IMIS  will  relate  to  BTS  by  improving  upon  the 
integration  of  training  records  (status  information  on  personnel).  All  training  status  information  would  be 
centrally  stored  on-line  where  relevant  personnel  would  have  access  to  it. 

Instructional  Desien 

It  is  possible  that  the  capabilities  of  the  APS  may  be  augmented  by  linking  it  with  the  Automated  Instructional 
Design  Advisor  (AIDA)  of  the  Advanced  Training  System  (ATS)  to  serve  the  purpose  of  designing,  developing, 
delivering,  and  evaluating  maintenance  training. 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 

This  report  describes  our  work  on  part  of  a  project  to  analyze  how  IMIS  can  be  used  in 
maintenance  training.  In  this  part  of  the  project,  our  goal  was  to  analyze  the  current  Air  Force 
maintenance  training  environment  and  develop  recommendations  and  hypotheses  for  how  IMIS 
could  be  used  in  this  environment.  In  order  to  do  this,  we  also  analyzed  the  current  Air  Force 
maintenance  process  and  the  capabilities  of  IMIS.  We  collected  data  on  maintenance  and  the 
maintenance  training  process  through  structured  interviews  at  Technical  School  (Lowry  AFB), 
FTD  (Hill  AFB),  and  OJT  (Hill  AFB,  Kelly  AFB)  sites.  We  also  investigated  IMIS's  capabilities 
by  attending  IMIS  demonstrations  and  examining  relevant  literature. 

Our  data  on  the  maintenance  process  suggests  that  Air  Force  maintenance  technicians  need 
training  in  a  large  number  of  procedures.  This  need  is  being  met  fairly  well  at  the  current  time 
through  the  existing  training  environment  and  the  system  of  TOs.  On  the  other  hand,  the  Air 
Force  also  has  a  critical  need  to  train  technicians  who  can  use  the  variety  of  problem-solving 
knowledge  required  for  expert  troubleshooting.  Many  of  the  maintenance  personnel  we 
interviewed  suggested  that,  currently,  this  need  is  not  being  met. 

Our  review  of  the  training  literature  suggests  that  cognitive  apprenticeship  training  is  a  very 
effective  way  to  teach  the  complex  problem-solving  skills  needed  for  expert  troubleshooting. 
This  approach  involves  realistic  problem-solving  practice,  coaching,  fading,  and  collaborative 
learning.  Our  data  on  the  maintenance  training  process  showed  that  some  aspects  of 
apprenticeship  training  are  used  in  Air  Force  OJT,  where  instructors  rely  on  demonstration, 
coaching,  and  practice  on  the  flightline  or  with  training  aircraft.  Training  in  the  classroom 
environments  (Technical  School  and  FTD)  relies  more  on  formal  methods  such  as  lectures  and 
written  tests.  However,  even  in  the  classroom  environments,  instructors  give  students  realistic 
practice  as  much  as  possible,  using  actual  aircraft  or  maintenance  simulators.  A  major  limitation 
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of  the  current  training  system  is  that  students  do  not  receive  enough  practice  on  maintenance 
tasks,  especially  difficult  troubleshooting  tasks  that  require  problem-solving.  This  limitation  is 
especially  evident  in  the  classroom  environments. 

So  far,  we  have  identified  a  need  for  troubleshooting/problem-solving  training,  a  method  for 
providing  that  training  via  coached  practice,  and  a  problem,  in  that  Air  Force  maintenance 
technicians  do  not  receive  enough  problem-solving  practice.  We  feel  that  IMIS  can  provide  the 
solution  to  this  training  problem  if  it  is  used  as  a  simulator  that  has  access  to  the  intelligence  and 
information  in  the  DM  and  the  CDM.  We  described  two  kinds  of  IMIS  simulators,  a  generic 
simulator  that  uses  primarily  the  knowledge  currently  in  the  DM  and  CDM,  and  a  detailed 
simulator  that  uses  enhanced  output  capability  (e.g.,  video)  and  authoring  of  additional 
knowledge  into  the  CDM. 

The  generic  simulator  would  use  all  the  principles  of  apprenticeship  training  and  explicitly 
teach  most  of  the  types  of  knowledge  required  for  expert  troubleshooting.  It  would  also  be  a 
relatively  low-cost  environment,  since  little  authoring  and  updating  of  instructional  materials 
would  be  needed.  The  key  factors  that  currently  hmit  IMIS's  usefulness  in  providing  training  are 
the  lack  of  human-like  mental-model  knowledge  and  coordination  processes  in  the  DM  and 
CDM.  Some  of  this  missing  troubleshooting  knowledge  possible  could  be  added  to  the  detailed 
IMIS  simulator.  Also,  a  videodisk  capability  would  allow  the  detailed  simulator  to  train 
procedures  with  significant  perceptual  components,  such  as  removing  and  replacing  parts. 

We  feel  that  both  of  these  IMIS  simulators  would  provide  powerful  training  environments 
that  would  effectively  teach  maintenance  skills,  including  critical  troubleshooting  skills.  Also, 
these  simulators  probably  would  easily  fit  into  the  current  training  practices  in  the  Technical 
Schools  and  FTD  classrooms,  given  the  value  placed  on  simulators  in  these  environments.  We 
have  also  outlined  ways  in  which  IMIS  could  fit  into  the  OJT  training  environment.  In 
conclusion,  the  training  capability  of  IMIS  has  the  potential  of  being  one  of  the  most  impressive 
aspects  of  this  system. 
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APPENDIX  A:  INTERVIEW  QUESTIONNAIRE 

(As  mentioned  previously  all  questionnaires  are  variations  of  one  another.  Therefore,  in  this 
Appendix  we  have  included  the  most  inclusive  questionnaire;  all  others  are  subsets.) 
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TRAINING  SITUATION  AND  TASK  EVALUATION  QUESTIONNAIRE 


This  questionnaire  was  developed  for  use  by  Mei  Technology  analysts  in  the  conduct  of 
structured  interviews  and  is  being  administered  in  conjunction  with  research  that  is  currently 
underway  at  Wright-Patterson  AFB,  Ohio  and  Brooks  AFB,  Texas.  This  questionnaire  has  a  dual 
data  gathering  purpose.  First,  it  will  assess  the  current  training  situation  for  flightline 
maintenance  performance  and  training  personnel  in  order  to  analyze  the  capabilities  of  IMIS  to 
support  maintenance  training.  Second,  it  will  assess  the  current  maintenance  training  provided  in 
the  formal  technical  school  and  help  determine  the  potential  impact  of  introducing  IMIS  into  that 
environment.  Your  expertise  in  this  area  is  greatly  appreciated  and  your  answers  are  strictly 
confidential. 

Thank  you  for  your  time  and  cooperation. 


PERSONNEL  PROFILE  FOR  MAINTENANCE  SUPERVISORS 

Please  complete  the  following  information: 

AFSC:  _  SKILL  LEVEL: _ 

Job  Title: _ 

Approximately  how  long  have  you  been  in  the  Air  Force? _ _ _ 

Approximately  how  long  have  you  been  in  the  maintenance  career  field? _ 

Approximately  how  long  have  you  been  in  this  specialty?  - - 

Approximately  how  long  have  you  been  in  your  current  work  center?  _ _ — 

Approximately  how  long  have  you  been  a  maintenance  supervisor? _ 

Approximately  how  long  have  you  been  an  OJT  trainer? - - - 

Please  describe  the  type  (i.e.,  actual  hands-on,  knowledge  of  operations,  etc.)  and  extent  of  IMIS 
experience  you  have  had  _ _ _ _ _ _ 

TASK  AND  TRAINING  INFORMATION 
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TASK#. 


In  reference  to  the  above  stated  task,  please  answer  the  following  questions: 

1 .  Is  this  task  currently  being  performed  in  this  specialty  and  this  workcenter? 
_ no  (continue  with  question  #  2) 

_ yes  (go  to  question  #  3) 

2.  Please  indicate  with  a  check  mark  which  of  the  following  is  true: 

_ ^The  task  has  been  combined  with  another,  (please  explain)  _ 


The  task  has  been  split  into  two  or  more  tasks,  (please  explain) 


_ ^The  task  is  no  longer  being  performed.  (Thank  you  for  your  time;  the  questionnaire 

for  this  task  is  complete). 

3.  Is  training  provided  for  this  task? 

_ yes  (continue  with  question  #  4) 

_ no  (go  to  question  #17) 

4.  How  is  the  trainee  pre-assessed  for  instruction? _ 


5.  How  is  training  conducted  for  this  task?  (please  indicate  all  those  that  apply) 
Initial/Upgrade  Training:  Qualification  Training: 

_ On  the  Job  Training  _ OJT 

_ Technical  School  _ Technical  School 

_ Field  Training  Detachment  _ FTD 

_ Aircraft  Maint.  Qual.  Program  _ AMQP 

_ Career  Development  Course  _ CDC 

6.  Who  typically  conducts  the  training  on  this  task?  (please  indicate  all  those  that  apply) 

Initial/Upgrade  Training:  Qualification  Training: 

_ supervisor  _ supervisor 

_ trainer  (non-supervisory)  _ trainer  (non-supervisoiy) 

_ FTD  instructor  _ FTD  instructor 

_ journeyman  technician  _ journeyman  technician 

_ Technical  School  instructor  _ Technical  School  instructor 

_ other  (please  explain) _ 


TASK  AND  TRAINING  INFORMATION  (continued) 

7a.  What  things  are  taught  during  initial/upgrade  training?  (please  indicate  all  those  that  apply) 
_ subtasks _ 
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equipment/tool  familiarization _ 

_ extenuating  flightline  procedures _ 

_ administrative  procedures _ ^ _ 

_ system/subsystem  knowledge _ 

_ other  _ _ 

7b.  What  things  are  taught  during  qualification  training?  (please  indicate  all  those  that  apply) 
_ subtasks _ _ _ 

_ equipment/tool  familiarization  _ _ 

_ extenuating  flightline  procedures  _ _ 

_ administrative  procedures _ 

_ system/subsystem  knowledge  _ 

_ other  _ _ 

8.  What  methods  are  used  for  conducting  the  training?  (please  indicate  all  that  apply) 

Initial/Upgrade  Training:  Qualification  Training: 

_ demonstration/practice  _ demonstration/practice 

_ oversight  and  assistance  _ oversight  and  assistance 

_ formal  training  _ formal  training 

_ independent  work  with  checks  _ independent  work  with  checks 

_ Computer  Based  Training  _ CBT 

_ other  (please  explain) _ 

9.  What  parts  of  the  training  are  self-paced? _ 


10.  What  is  the  approximate  average  duration  (from  beginning  to  end)  of  training  on  this  task? 

Initial/Upgrade: _ _ _ _ _ 

Qualification: _ _ _ _ _ 

1 1.  How  often  is  training  provided  (e.g.,  every  day,  once  a  week,  every  two  weeks,  etc.)? 

Initial/Upgrade: _ _ _ _ _ 

Qualification: _ _ _ _ _ 

12.  Approximately  how  long  is  each  training  session? 

Initial/Upgrade: _ _ _ - — 

Qualification:  _ _ _ _ _ _ 


13.  How  are  students  evaluated?  (please  indicate  all  that  apply) 

Initial/Upgrade  Training:  Qualification  Training: 

_ through  use  of  a  checklist  _ through  use  of  a  checklist 

_ written  tests  _ written  tests 
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_ part  task  trainers  or  mock-ups  _ part  task  trainers  or  mock-ups 

_ actual  equipment  performance  _ actual  equipment  performance 

_ other; _  _ other: _ 

14.  Who  conducts  the  evaluation?  (please  indicate  all  that  apply) 

Initial/Upgrade  Training:  Qualification  Training: 

_ supervisor  _ supervisor 

_ Quality  Control  personnel  _ Quality  Control  personnel 

_ other: _ _  _ other: _ _ 

15.  How  is  task  performance  certified/decertified,  and  by  whom? 

Initial/Upgrade:  _ _ 


Qualification: 


16.  Please  describe  the  recordkeeping  process. 


TASK  CHARACTERISTICS 

17.  Please  answer  the  following  questions  with  "y"  for  "yes"  and  "n"  for  "no": 

_  Are  the  task  performance  requirements  hard  to  communicate  through  printed  words? 

_ Would  the  performance  of  this  task  benefit  from  the  use  of  graphs,  charts,  tables, 

or  illustrations? 

_ Does  the  learning  of  this  task  prepare  the  individual  to  perform  a  subsequent  task? 

_ Does  the  performance  of  this  task  require  the  verification  of  readings  and  tolerances? 

_ Does  the  performance  of  this  task  require  high  accuracy  and  completeness  due  to  the 

presence  of  high  risk  or  danger? 

_ Is  this  task  too  complex  to  learn  on  the  job? 

TASK  CHARACTERISTICS  (continued) 

_ Does  this  task  require  long  and  complex  behavioral  sequences? 

_ Does  this  task  require  many  decisions? 

_ Is  this  task  performed  in  a  fixed  sequence  or  order? 
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_ Is  this  task  so  new  or  undefined  that  the  detailed  steps  are  hard  to  master? 

_ Does  this  task  consist  of  steps  performed  in  rapid  succession? 

_ Does  this  task  require  rapid  response  such  that  there  is  no  time  to  refer  to  printed 

instruction? 

_ Is  this  a  lengthy  task  that  requires  attention  to  detail? 

_ Is  this  task  frequently  performed  (perhaps  several  times  a  day)? 

_ Is  this  task  performed  by  a  large  number  of  the  individuals  in  this  specialty? 

_ Does  this  task  need  much  practice  for  acceptable  performance? 

_ Is  reference  to  printed  material  disruptive  to  performance  of  this  task? 

_ Does  the  performance  of  this  task  require  making  decisions  based  on  an  awareness  of 

numerous  existing  conditions? 

_ Does  the  performance  of  this  task  require  branching  logic  such  as  that  found  in 

diagnostic  decision  aids? 

_ Does  the  performance  of  this  task  require  team  effort? 

_ Does  the  performance  of  this  task  require  only  one  person  to  perform? 

_ Are  the  consequences  of  task  performance  errors  serious,  such  as  with  emergency 

procedures? 

_ Is  an  occasional  performance  error  likely  to  result  in  equipment  damage? 

_ Is  the  risk  of  damage  to  personnel  or  equipment  high? 

_ Is  the  task  costly  to  train? 

_ Are  both  training  time  and  budget  limited? 

TASK  CHARACTERISTICS  (continued) 

_ Is  memorization  (of  steps  involved)  required  for  performance  of  this  task? 

_ Can  the  task  be  mentally  rehearsed  before  it  needs  to  be  performed? 

_ Is  the  task  likely  to  change  frequently? 

_ Is  the  performance  method  likely  to  change? 
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Is  maintenance  equipment  easily  accessible? 

Is  maintenance  equipment  usually  available? 

Does  the  task  require  detailed  procedures  to  properly  access  equipment? 


SUPPLEMENTARY  QUESTIONNAIRE:  MAINTENANCE  SUPERVISORS 

Troubleshooting  Information  :  The  information  requested  for  this  questionnaire  is  not  specific 
to  any  task  in  particular,  rather  it  concerns  troubleshooting  in  general. 

A.  Types  of  faults  encountered. 

We  are  interested  in  receiving  information  that  is  specific  to  troubleshooting  tasks  performed 
by  the  maintenance  technicians  you  supervise.  By  troubleshooting,  we  mean  diagnosing  and 
fixing  faults  in  a  system.  We  will  define  two  types  of  faults,  "technical-order"  faults,  and 
"problem-solving"  faults. 

Technical -order  faults  can  be  handled  simply  by  following  the  proper  technical  orders  and 
applying  basic  knowledge  about  the  system  and  about  how  to  perform  tests  and  repairs.  The 
key  characteristic  of  a  technical-order  fault  is  that  the  technician  does  not  have  to  make  any 
major  decisions  about  the  what  tests  or  repairs  to  make  or  how  to  sequence  tests  or  repairs. 
All  of  these  decisions  are  made  by  the  technical  orders. 

In  problem-solving  faults,  on  the  other  hand,  the  technical  orders  do  not  provide  enough 
information  to  diagnose  or  repair  the  fault.  Here  the  technician  must  make  major  decisions 
about  what  tests  or  repairs  to  make  and/or  how  to  sequence  these  tests  and  repairs. 

1 .  Estimate  the  percentage  of  technical-order  faults  that  your  maintenance  technicians 

encounter _ % 

2.  Estimate  the  percentage  of  problem-solving  faults  that  your  maintenance  technicians 

encounter _ % 

B.  Current  troubleshooting  practices. 

We  have  listed  a  number  of  approaches  to  troubleshooting  below.  For  the  average 
maintenance  technician  you  supervise,  please  estimate  the  percentage  of  troubleshooting 
tasks  performed  using  each  of  these  approaches.  Since  technicians  may  use  more  than  one 
approach  on  a  single  problem,  the  percentages  do  not  have  to  add  to  100%. 

On  what  percent  of  the  problems  encountered,  does  the  average  technician: 

a.  follow  technical  orders? _ % 
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b.  recall  how  he/she  solved  the  same  problem  before  and  use  this  procedure? 

_ _ _% 

c.  recall  how  he/she  solved  a  similar  problem  before  and  use  this  as  an  analogy 

or  example? _ % 

d_.  ask  co-workers  for  help? _ % 

e.  use  troubleshooting  strategies? _ % 

examples  of  strategies: 

-working  backwards  from  the  bad  output  or  symptom 
-eliminating  components  you  know  to  be  working 
-half  split  (i.e.,  choosing  a  test  that  will  eliminate  about  half  of  the 
possibly  faulty  components) 

f.  other  approaches? _ %  (Please  explain)  _ 


C.  How  troubleshooting  is  taught. 

1 .  The  following  are  different  kinds  of  knowledge  that  are  used  in 
troubleshooting  (diagnosis  and  repair).  Use  the  5-point  scale  shown  below  to 
indicate  how  extensively  each  kind  of  knowledge  is  covered  in  training. 


no 

moderate 

extensive 

coverage 

coverage 

coverage 

1 

2 

3 

4  5 

a.  _ device,  or  system,  knowledge  (i.e.,  information  that  is  specific  to 

various  aircraft  subsystems  that  are  maintained) 

b.  _ procedural  knowledge  (i.e.,  specific  steps  involved  in  diagnosis  and 

repair  of  the  aircraft) 

c.  _ schematics  or  visual  maps  of  how  the  various  parts  of  the  sub¬ 

system  relate  and  fit  together,  and  how  they  fit  into  the  scheme  of 
the  entire  aircraft 

d.  _ symptom-fault  associations  (i.e.,  how  to  find  frequently  occurring 

faults  at  a  glance  by  recognizing  the  pattern  of  symptoms,  or  bad  outputs, 
caused  by  the  faults) 
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e.  _ troubleshooting  strategies  (e.g.,  half-split;  work  backwards  from  the 

bad  output  or  symptom;  eliminate  components  that  are  known  to  be 
working) 

f.  _ specific  test  procedures  (e.g.,  using  an  oscilloscope) 

g.  _ time  and  cost  considerations  (i.e.,  advantages  and  disadvantages  of 

performing  various  tests  and  repairs) 

h.  _ other: _ 


2.  What  methods  are  used  to  teach  troubleshooting?  (please  check  all  that  apply) 

a.  _ demonstration/practice 

b.  _ oversight  and  assistance 

c.  _ formal  training 

d.  _ independent  work  with  checks 

e.  _ CAI/CBT 

f.  _ other  (please  explain)  _ 


3.  Approximately  what  percentage  of  training  focuses  explicitly  on  the  aspects  of 
troubleshooting  that  require  problem  solving  (e.g.,  using  strategies  such  as  half 

split  and  elimination,  considering  both  the  costs  and  the  information  gained 
when  deciding  on  a  test)? _ % 

4.  Approximately  what  percentage  of  training  focuses  explicitly  on  aspects  of 

troubleshooting  that  do  not  require  problem  solving  (e.g.,  following  technical 
orders,  how  to  use  test  equipment  and  make  repairs)? _ % 


IMIS  Information 

D.  Effects  of  IMIS  on  performance  and  training. 

Given  what  you  know,  or  have  been  told  about  IMIS: 
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1.  In  what  ways  do  you  feel  the  introduction  of  IMIS  to  flightline  maintenance 
will  impact  the  technician's  performance  of  his/her  job? _ 


2.  In  what  ways  do  you  feel  the  availability  of  this  system  will  affect  maintenance 


APPENDIX  B:  PREREQUISITES  FOR  SELECTING  INTERVIEWEES 
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Personnel  Type 


Maintenance 

Supervisor 


Maintenance 

Technicians 


Maintenance  Training 
Manager(s) 


Selection  Criteria 

Conditions 

1.  Three  first  line  supervisors  to  be  interviewed  in 
each  of  the  three  AFSs  (total  of  nine) 

Desired 

Number 

2.  Each  supervisor  to  have  knowledge  of  current 
flightline  maintenance  training  procedures,  to 
include  conduct  of  training,  evaluation,  and  record 
keeping. 

Required 

3.  Each  supervisor  to  have  general  knowledge  of 
IMIS  operations  as  they  relate  to  flightline 
procedures. 

Desired 

4.  Each  supervisor  to  have  general  knowledge  of 
IMIS  operations  as  they  relate  to  specific  task 
performance  of  those  tasks  chosen  in  each  AFS. 

Desired 

5.  Each  supervisor  to  be  skilled  in  the 
performance  and  training  of  the  tasks  chosen  in 
each  AFS. 

Required 

1.  Three  technicians  to  be  interviewed  in  each  of 
the  three  AFSs  (total  of  nine) 

Desired 

Number 

2.  Each  technician  to  have  general  knowledge  of 
IMIS  operations  as  they  relate  to  the  flightline 
maintenance  process. 

Desired 

3.  Each  technician  to  have  specific  knowledge 
and  experience  with  IMIS  operations  as  they  relate 
to  specific  task  performance  for  the  chosen  tasks 
in  each  AFS. 

Desired 

4.  Each  technician  to  be  skilled  in  the 
performance  of  the  chosen  tasks  in  each  AFS. 

Required 

1.  One  or  more  maintenance  training  manager(s) 
having  training  management  responsibility  for 
each  of  the  three  AFSs. 

Required 

2.  Each  maintenance  training  manager  to  have 
knowledge  of  the  flightline  maintenance  training 
and  record  keeping  process. 

Required 
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Personnel  Type 


Field  Training 
Detachment  (FTD) 
Instructor(s) 


Technical  Training 
Instructor  (s) 


Course  Supervisor(s) 


Selection  Criteria 

Conditions 

1.  At  least  one  Field  Training  Detachment 
Instructor  in  each  of  the  three  AFSs  who  has 
current  experience  formally  teaching  associated 
tasks 

Required 

2.  Each  FTD  instructor  to  have  knowledge  of 
current  FTD  maintenance  training/instructional 
procedures  to  include  conduct  of  instruction, 
evaluation,  and  record  keeping 

Required 

3.  Each  FTD  instructor  to  have  general 
knowledge  of  the  flightline  maintenance  process 
to  include  postflight,  debriefing,  scheduling, 
troubleshooting,  parts  ordering,  repair,  and  status 
reporting. 

Desired 

4.  Each  FTD  instructor  to  be  skilled  in  the 
performance  and  training  of  the  chosen  tasks  in 
their  AFS. 

Preferred 

1.  At  least  one  instructor  in  each  of  the  three 

AFSs  who  has  current  experience  formally 
teaching  the  associated  tasks. 

Required 

2.  Each  instructor  to  have  knowledge  of  current 
technical  school  maintenance  instructional 
procedures  to  include  conduct  of  instruction, 
evaluation,  and  record  keeping. 

Desired 

3.  Each  instructor  to  have  general  knowledge  of 
the  flightline  maintenance  process  to  include 
postflight,  debriefing,  scheduling,  troubleshooting, 
parts  ordering,  repair,  and  status  reporting. 

Desired 

4.  Each  instructor  to  be  skilled  in  the  performance 
and  training  of  the  tasks  for  their  AFS. 

Preferred 

1.  At  least  one  course  supervisor  to  have 
managerial  and  administrative  responsibility  for 
AFS  courses  which  teach  the  chosen  tasks  for  that 
specialty. 

Required 

2.  Each  course  supervisor  to  have  knowledge  of 
the  course  development  process. 

Required 
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